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ABSTRACT 



This thesis provides a comparative analysis of ship Operational Test and 
Evaluation (OT&E) in the United States Navy (USN) and the Royal Australian 
Navy (RAN). It also reviews the acceptance and introduction into service 
procedures for warships in both Navies, including the input from OT&E. This study 
analyses USN and RAN Ship OT&E organisation, policy, and procedures, and then 
compares and contrasts the two systems. The study finds that the RAN OT&E 
system, although originally based on USN OT&E philosophy, now differs in the 
importance, interpretation, application and focus of OT&E. It concludes that to 
achieve efficient and effective trials and acceptance of the new higher risk warships 
currently under construction, the RAN OT&E system needs to be revised. A model 
for OT&E in the RAN is proposed based on the principles derived from the USN 
system. The recommendations include the initial conduct of OT&E in land based 
test sites, followed by dedicated “Whole Ship” OT&E for the first of class. 
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EXECUTIVE SUMMARY 



With new frigates and submarines under construction, the Royal Australian 
Navy (RAN) has embarked on its greatest combatant shipbuilding program since the 
Second World War. Although these combatants are being built to proven designs 
and, therefore, are considered relatively low risk, they do present a greater 
operational risk than recent RAN combatant programs. In September 1991, the 
Chief of Naval Staff (CNS) expressed concern that the RAN may not be capable of 
adequately managing the comprehensive and complex procedures necessary to 
accept the ANZAC ships and Collins class submarines into service. This focussed 
attention on the acceptance procedures, and consequently on the Operational Test 
and Evaluation (OT&E), of the new combatants. The United States Navy (USN) is 
the world leader in the development of high technology weapons systems and have 
extensive experience in their acquisition, testing and operation. As the RAN has a 
similar philosophical approach to acquisition and Test and Evaluation (T&E), the 
USN OT&E system forms a credible basis from which to develop an RAN OT&E 
organisation, policies and procedures capable of adequately assessing the 
operational effectiveness and operational suitability of the new combatants against 
the user requirement. 

A. SHIP TEST AND EVALUATION 

The fundamental purpose of T&E in a system’s development and acquisition 
program is to identify the areas of technical risk to be reduced or eliminated. During 
the early phases of development, T&E is conducted to demonstrate the feasibility of 
conceptual approaches, to minimise design risk, to identify design alternatives, to 
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compare and analyse trade-offs, and to estimate operational effectiveness and 
suitability. As a system undergoes design and development, the emphasis in testing 
moves gradually from developmental test and evaluation, which is concerned 
chiefly with the attainment of engineering design goals, to operational test and 
evaluation, which focuses on questions of operational effectiveness, suitability and 
supportability. The principal types of T&E pertaining to ships as defined by the USN 
are: 

1. Developmental Test and Evaluation (DT&E) 

DT&E is conducted to assist the engineering design and development 
process and to verify attainment of technical performance specifications and 
objectives. 

2. Production Acceptance Test and Evaluation (PAT&E) 

PAT&E is testing conducted on production items to ensure systems meet 
contract specifications and requirements, usually for contractual acceptance 
purposes. It is a type of DT&E. 

3. Operational Test and Evaluation (OT&E) 

OT&E is conducted to determine a system's operational effectiveness and 
operational suitability, identify system deficiencies and the need for potential 
modifications to meet established OT thresholds, and develop tactics. OT&E has 
three distinguishing characteristics: 

• It is conducted in an operationally representative environment. 

• It is conducted on production representative equipment using fleet personnel 
for operation and maintenance. 

• It is conducted against a threat-representative simulated enemy carrying out 
threat tactics per the latest threat assessment. 
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The focus of DT&E is on a system meeting technical and production 
specifications, while in OT&E, the focus is on assessing the actual functioning of the 
system in the realistic combat environment, against the user requirement. 

B. OT&E ENVIRONMENT 

The U.S. places a high priority on Defence with strategies of world leadership, 
global influence and self sufficiency. To achieve these strategies, the U.S. devotes a 
high percentage of its Federal expenditure to Defence, and places a high priority on 
Research and Development (R&D). The USN need for OT&E stems from its almost 
exclusive reliance on indigenous weapons development and production. It has a 
history of large, risky, developmental programs which push the state of the art in 
weapons technology. To minimise technical and operational risk, the outputs of 
these programs require the assessment of operational effectiveness and suitability 
against the user requirement before being committed to production and subsequent 
introduction into the Fleet. 

Australia’s Defence strategy has progressed from a position of dependence on 
allies, to a positive acceptance of both self-reliance and regional influence. To 
achieve its Defence goals, however, Australia devotes much less of its resources and 
places less priority on R&D than does the U.S. The need for local T&E is recognised 
by the Australian Government and DoD, for both indigenous and overseas systems, 
however, the importance of OT&E in particular, is not specifically addressed in any 
Australian T&E policy document. This lack of recognition of the importance of 
OT&E is perhaps due to the RAN procuring low risk, complete ships in the past 
(e.g., FFG-7 class) with the wealth of USN DT&E and OT&E behind it. 
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C. OT&E IN THE ACQUISITON PROCESS 



The U.S. Defence system acquisition life cycle consists of progressive 
development phases separated by major decision milestones, when a program is 
reviewed and authorised to advance to the next stage. The USN OT&E system forms 
an integral part of the U.S. DoD acquisition process. It is an empirical method of 
ensuring a sufficient technical return on acquisition investment and for ensuring that 
a new system is fully capable of meeting the fleet’s needs. As a result, the U.S. place 
high importance on OT&E, with its conduct being mandated by Congress and 
incorporated in the law of the U.S. OT&E is viewed as being more important as a 
basis for a decision to proceed beyond Low Rate Initial Production, than on the 
introduction into service of the final production item. This is because the “big 
bucks" of most Defence acquisitions are spent during the production phase. OT&E, 
however, has started to play an important role also in the assessment of first of class 
systems for introduction into the USN Fleet. 

The USN and RAN have similar acquisition systems, however, the level of 
importance of T&E, and OT&E in particular, differs greatly. The Australian DoD 
prefers lower risk, proven technologies for the majority of new acquisitions. The 
Australian system, therefore, places a high T&E priority on the post production 
phase, with no mandatory requirement for OT&E prior to a production or purchase 
decision, but requiring OT&E before the first production system is Accepted into 
Naval Service (AINS). As Australia is now buying far less “off the shelf’ systems 
from overseas, and systems are increasingly being designed and built in Australia or 
adapted from overseas designs, the operational risk of these systems is increasing. 
These greater risks place emphasis on OT&E leading to a production or purchase 
decision where the majority of a project’s funding is spent. 
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D. OT&E ORGANISATION 

The organisation of OT&E within the USN is complex with separate authorities 
performing a number of OT&E functions, as listed in Table 1 . 



TABLE 1 : USN OT&E FUNCTIONAL AUTHORITIES 



OT&E Function 


USN Authority 


Policy Maker & Overseer 


Director, Operational Test & Evaluation 
(DoD authority) 


Sponsor 


Chief of Naval Operations 


Developing Agency 


Commander, 

Naval Sea Systems Command 


Operational Tester & Evaluator 


Commander, 

Operational Test & Evaluation Force 


User 


Fleet Commanders, 

Ships’ Commanding Officers 


Coordinator 


Director, Test, Evaluation and Technology 
Requirements 



Significant characteristics of this organisation include the appointment of a 
Director of OT& E (DOT&E) responsible for policy formulation, evaluation and 
oversight of OT&E within the U.S. DoD, and the inclusion of a coordinator for USN 
T&E. The USN also separates the OT&E and DT&E organisation structures. 

Following a number of recent reorganisations, the RAN’s OT&E structure is in 
disarray. There is no clear OT&E, or T&E, policy maker or overseer within the 
Australian DoD or the RAN. The Sponsor is not within the RAN organisation, being 
part of centralised force development and user requirements organisation (HQADF) 
who sponsors major RAN acquisitions, and is responsible for the initial 
determination of T&E requirements. The Developing Agency (ACMAT-N), unlike 
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its USN counterpart, does not have a T&E office to support Project Directors. With 
the disestablishment of the RAN Trials and Assessing Unit (RANTAU) in May 
1992, the RAN’s OT&E authority is now the Commander, Test and Evaluation 
(CTE) under the Chief Staff Officer (Engineering) (CSO(E)) in the Maritime 
Command. CTE is currently multifunctional, conducting DT&E and PAT&E in 
addition to OT&E. This results in a lack of differentiation within CTE's organisation 
between DT&E/PAT&E and OT&E. Also being placed under the in-service 
engineering area of the Maritime Command, he lacks visibility and influence within 
the warfare community and hence credibility in operational matters critical to 
OT&E. CTE’s OT&E responsibilities are overshadowed by his DT&E/PAT&E 
duties. The RAN currently has no single authority who acts as a T&E focal point 
within Navy Office responsible for coordination of T&E matters, however, a project 
T&E manager does perform this function for his project. 

In summary, the RAN and Australian DoD have reorganised over the past five 
years to create a more effective and efficient user requirements organisation, and to 
better accord with Program Management and Budgeting principles. However, as a 
result, the RAN has no T&E, or OT&E policy maker and overseer, no OT&E 
coordinator within Navy Office, and the RAN’s OT&E authority is buried within the 
in-service engineering management area of the Maritime Command. 

E. OT&E POLICY 

Where USN OT&E policy has its basis in the statutes approved by Congress, 
the RAN has no such formal basis and its OT&E policy is embedded in the more 
general T&E policy. Some key aspects of this policy include: 
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1. OT&E Policy Documentation 

The USN has a hierarchy of comprehensive OT&E policy documentation 
that provides clear, consistent and non-conflicting policies and guidance to OT&E 
participants, and establishes the disciplined management approach to OT&E taken 
by the USN. In contrast, the current RAN T&E policy documentation provides 
broad T&E policy only and, other than defining OT&E, provides little policy to 
guide its conduct. The RAN has adapted the more detailed USN T&E definitions to 
suit its own requirements. During this process, however, the RAN lost the 
distinction between the types of T&E, and often confuses OT&E with PAT&E/ 
DT&E. Overall, RAN T&E documentation is not comprehensive, inconsistent and 
fails to provide adequate guidance. 

2. Independence of OT&E Authorities 

Independence of OT&E authorities is the key to the effectiveness of OT&E 
in the U.S. acquisition process. OPTEVFOR, as the independent OT&E agency 
within the USN, reports directly to CNO, while DOT&E provides independent 
oversight and coordination of the military services’ planning and execution of 
OT&E, reporting directly to the Secretary of Defence. The independence of the 
OT&E authorities is designed to ensure impartiality, and honest and open reporting, 
subject to the minimum of political interference. 

Although major acquisition decisions within the RAN are less based on the 
results of OT&E, the RAN also maintains the policy that OT&E is to be conducted 
by an authority independent of the development and production agencies. CTE, on 
the staff of the Maritime Commander, is independent from the development and 
production agencies, but not from the end user, as in the USN. The implications of 
this lack of independence is that the end user may have his own aims and objectives 
which conflict with the total impartiality of OT&E conduct and reporting. 
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3. Foreign Weapons Evaluation 

The USN has a foreign weapons evaluation program which is designed to 
support the evaluation of foreign weapons systems, equipment or technology in 
terms of its potential to meet specific U.S. military requirements. The primary 
objective of the program is to reduce the costs of research and development, while 
leading to the acquisition of foreign equipment for U.S. use. Despite the most recent 
Australian Government “White Paper” on Defence Policy requiring the need to be 
able to determine the performance in the Australian environment of equipment of 
both overseas and local origin, no DoD or RAN T&E policy specifically addresses 
the evaluation of foreign systems. 

4. Land Based Test Sites (LBTS) 

Although used primarily for the purpose of conducting DT&E, the USN has 
demonstrated that early OT&E on LBTS can give an estimation of potential 
operational effectiveness and suitability, and hence identify potential operational 
problems early and minimise operational risk. RAN guidance for OT&E using 
LBTS includes a statement that system centres and simulators will be employed for 
early stages of OT&E if available, and that OT-2 may be conducted at a land-based 
test site. The use of LBTS for OT&E during the ANZAC frigate and COLLINS 
submarine projects would appear to offer advantages to the RAN. However OT-2 in 
support of a production decision was not conducted for either the ANZAC frigate 
and Collins submarine project. Although LBTS are being developed for both 
projects, they were not available before production contracts for all the required 
combatants were signed. The use of these LBTS for early OT-3 also was not planned 
in the original project schedules, and so the conduct of OT&E is now subject to them 
being used on a non-interfering basis to the contractor. LBTS should be used for 
OT&E in the RAN to give an estimation of potential operational effectiveness and 
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suitability, and hence identify potential operational problems early and minimise 
operational risk. This should be conducted as the first phase of OT-3 during a 
dedicated period following the completion of DT&E on the LBTS. 

5. Clear Delineation between T&E Types 

The USN achieves a clear delineation between the different types of T&E. 
OPTEVFOR conduct only OT&E, and OT&E within the USN is conducted solely 
by OPTEVFOR. Thus, there appears to be good understanding in the USN of what 
is, and what isn’t, OT&E. USN policy also states that combined DT&E/PAT&E and 
OT&E testing should only be considered when there are time and cost savings. In 
addition, this combined approach must not compromise either the developmental or 
operational test objectives, and separate evaluations and reports will be prepared by 
DT&E and OT&E testers. The conduct of combined and concurrent DT&E. 
(PAT&E) and OT&E is not addressed in RAN T&E policy, however, it is implied as 
being almost a requirement. The RAN does not clearly differentiate between the 
types of T&E, which leads to DT&E and OT&E often being conducted over the 
same period, by the same test team. Although the RAN recognises, by definition, the 
difference in objectives and methodology between the types of T&E, no limitations 
or guidance as to the possible hazards of this combined testing approach are 
addressed. 

F. SHIP ACCEPTANCE 

The purpose of ship acceptance in the USN is to ensure delivery to the Fleet of 
complete ships, free from both contractor and government responsible deficiencies. 
Independent verification of readiness of ships for acceptance is the responsibility of 
the President, Board of Inspection and Survey (PREINSURV) who conducts 
Acceptance Trials and Final Contract Trials for each individual new ship. Ship 
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acceptance has traditionally been based only on the successful completion of 
PAT&E and material inspections that, for the first of class, culminated in approval 
for Fleet Introduction. However, with the treatment of ships as a complete weapons 
system, as for the DDG-51, Fleet Introduction is now based also on OT&E results. 

In the RAN, Delivery (contractual acceptance) of the ship to the Government is 
a contractual matter and is managed by the Project Director. This is followed by 
RAN trials to assess the ship for Acceptance into Naval Service (AINS). CTE’s 
responsibility is to conduct the trials and report the results. An Acceptance Board 
(AB) is established to provide an assessment of the ship and submit 
recommendations upon which CNS can base his acceptance decision. The President, 
Vice-President and Board members serve on the AB part time, and the sole function 
of the AB is to advise CNS on acceptance matters. It does not conduct tests, trials or 
inspections, and is essentially a board of review. Similar to the USN, the RAN also 
places emphasis on PAT&E leading to AINS, although OT-3 has been included in 
the trials program. However, the trials conducted as OT-3 were really PAT&E / 
DT&E with minimal true OT&E content. For the first of class, OT-4 is conducted 
following AINS. It comprises a series of trials and evaluations over a 12 to 24 month 
period to assess operational effectiveness and operational suitability leading to 
Operational Acceptance. 

The AB, with its broader experience and knowledge than CTE, is generally 
agreed within the RAN as being successful as a board of review of the planning and 
results of T&E for individual projects, and recommending the first of class for 
AINS. As an ad-hoc organisation consisting of various functional specialists 
brought together part time, however, the AB suffers from a number of problems. It 
requires guidance as to its role and functions, T&E philosophy and the procedures 
of the acceptance process. Although the AB attempts to obtain sufficient guidance 
from the T&E documentation, it has found this documentation to be unclear and 
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conflicting. With only DT&E/PAT&E and the minimal operational aspects of the 
OT-3 testing conducted by the RAN, the AB has tended to focus on materiel 
deficiencies, rather than the performance of the first of class against the user 
requirement. 

G. OT&E PROCEDURES 

1. Level and Extent of OT&E 

The determination of the level and extent of OT&E required is a difficult 
question. The fundamental cornerstone of OT&E is the user requirement, generated 
through a series of documents built into the acquisition process. A requirement can 
be traced from the capability need through the refining requirements documentation 
to a specific objective in the TEMP, to the test plan, and ultimately to the final OT&E 
report. This linkage from capability need to operational test program defines the 
scope of testing necessary to evaluate the final product against the current user 
requirement. 

2. Assessing Operational Effectiveness 

Operational effectiveness is the capability of a system to perform its 
intended function effectively over the expected range of operational circumstances, 
in the expected environment, and in the face of the expected threat, including 
countermeasures where appropriate. The USN examines each Critical Operational 
Issue (COI) and decides what needs to be known to enable each issue to be assessed. 
It ensures that the appropriate environments, threats, etc. are included and that 
sufficient data will be generated to address the COI and objectives. It focuses on 
achieving statistical relevance and making the tests as objective as possible. The 
broad scope of many operational requirements, however, makes their testing and 
subsequent assessment rather subjective, making the determination of meaningful 
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and assessable quantitative Measures of Effectiveness (MOEs) difficult. In addition, 
a lack of resources may reduce an objective, quantitative, statistically relevant test 
schedule to a more subjective, qualitative assessment. These situations require 
expertise and judgement, and compromise between the authorities involved, to 
make the tests as objective as possible. 

The RAN has found it difficult to define exactly what is required to assess 
operational effectiveness. The RAN recognises that the user requirement is the 
bench mark for determining the degree to which a system is effective, and also that 
operational effectiveness is best assessed by a performance demonstration by 
normal operating personnel in the normal or given environment. Assessing 
operational effectiveness has involved analysing each COI and then employing 
modified Ship Qualification Trial or Fleet Exercise Program techniques to evaluate 
them. The RAN has very little guidance on the assessment of operational 
effectiveness. 

3. Assessing Operational Suitability 

Operational suitability is the capability of a system, when operated and 
maintained by typical fleet personnel in the expected numbers and of the expected 
experience level, to be reliable, maintainable, operationally available, logistically 
supportable when deployed, compatible, interoperable and safe. The assessment of 
operational suitability in the USN consists of an Integrated Logistic Support (ILS) 
Plan review, followed by suitability testing based on the expected reliability, degree 
of confidence, thresholds, etc., required to be achieved. OPTEVFOR has analysts 
who design suitability tests and determine measures of suitability, and also evaluate 
and analyse the adequacy of logistic supportability. OPTEVFOR also relies on 
operational personnel to use their experience and knowledge of the system to 
identify inadequate logistic support. 
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Before 1987, the assessment of operational suitability, and ILS in 
particular, was rather subjective and often controversial in the RAN. A new 
approach was taken in 1987 to make the assessment of ILS more objective. The 
RAN is currently developing a routine in-service RM&A data collection and 
analysis system, however, until that is operational, the data is collected and analysed 
on an ‘as required’ basis by CTE. 

In summary, the principles of operational suitability assessment are 
common between the USN and RAN. Although assessments by the RAN have 
become more subjective over recent years, the assessment of RM&A is still in its 
infancy. The RAN requires experienced personnel to design suitability tests and 
analyse results. 

4. OT&E Planning 

The RAN has embraced the U.S. TEMP concept as the single executive 
document for the management of T&E for major acquisitions. Although required 
early in a project, recent RAN TEMPs have not been raised until after project 
funding has been approved. TEMPS need to be approved earlier to enable inclusion 
of post delivery PAT&E/DT&E and OT-3 in the Project budget, and OT-4 in the 
Maritime Commander’s budget. The authority for DCNS to release the TEMP was 
based on his role as Project Sponsor. With HQADF now assuming this role, the 
DCNS function in TEMP development is unclear. The TEMP approval process 
requires review. 

The USN T&E Coordinator responsibilities include the chairmanship of the 
Test and Evaluation Coordination Group (TECG) for each major program. Some of 
the functions of a TECG are the early definition of terms, measures of effectiveness, 
and the acceptability criteria. In the RAN, a T&E Planning Group (TEPG) is formed 
prior to TEMP development to analyse T&E requirements and estimate the 
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resources required. The TEPG is similar to the USN TECG with one significant 
difference. The USN TECG is chaired by the USN T&E Coordinator under the 
CNO, whereas the RAN TEPG is chaired by each individual Project Director 
through his T&E manager. Despite their best intentions. Project Directors are 
essentially driven by cost and schedule considerations, not the overall T&E 
adequacy of their project. To be truly objective, each TEPG should be chaired by the 
authority responsible for ensuring the adequacy of the RAN’s overall T&E program. 

5. RAN OT&E Reporting 

Reports provide the OT&E authority's conclusions regarding a system's 
operational effectiveness and suitability, and his recommendations regarding the 
systems future. The USN requires OT&E reports to be impartial, complete, 
thorough and be reported solely by the OT&E authority. 

Within the RAN, the OT&E Authority coordinates the issue of trials 
reports, however, they are issued on completion of each segment of OT&E e.g., o/c 
ASW section of OT-3 etc., usually in addition to a report at the end of the OT phase. 
Quicklook reports are also routinely sent on completion of each week’s testing. The 
ship under test also provides a report, however, it is usually forwarded to its 
operational authority and the Project, in addition to the OT&E Authority. This early 
dissemination of OT&E results, before a full analysis is complete and the 
implications assessed, can lead to other authorities taking hasty action on 
incomplete information, and can prejudice the OT&E authority’s final conclusions 
and recommendations. OT&E results should indicate system deficiencies against 
user requirements rather than equipment defects, so the results need to be fully 
analysed and the implications assessed before being reported to a wider audience. 
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6. Whole Ship OT&E 

The OT-III conducted on DDG-51 was the first time the USN has 
conducted OT&E on a whole ship in a multi-battle situation, in a free play 
environment. The whole ship was viewed as an integrated weapons system, rather 
than as a platform for weapons systems, and OT-III was conducted in a dedicated 
period following Ship Qualification Trials (SQT) and Final Contract Trials. 

Within the RAN, the current OT&E to support an AINS decision (OT-3) is 
little more than an extended SQT period where DT&E/PAT&E is conducted with 
some operational assessment. It is not considered to be adequate to assess the first 
of a new ship class against the user requirement. To assess the operational 
effectiveness and suitability of a ship against the user requirement with the degree 
of confidence required for the Acceptance Board to support an AINS decision, a 
dedicated, free play, scenario based, OT-3 period is required. 

a. OT&E Training 

OT&E is a specialised discipline, with its own philosophy and 
methodology, and so requires a specialist approach with knowledge and experience 
to make it effective. COMOPTEVFOR conducts a three day Operational Test 
Directors’ (OTD) course covering the major areas of OT&E. They also run irregular 
segment courses which provide overviews or updates on OT&E subjects e.g., 
analysis, test plan development and threat updates. 

The RAN has no training courses on OT&E, or on T&E in general. A 
number of officers have completed the USN OTD course in recent years which has 
improved the knowledge and understanding of OT&E within CTE’s organisation 
considerably. These courses have been arranged on an ad-hoc basis through 
overseas visit submissions rather than as pre-requisite courses for particular billets. 
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To be effective, OT&E requires experienced and knowledgable 
personnel with high professional credibility within their field of expertise. To 
achieve this requires selection of suitable personnel, adequate training and good 
guidance documentation, and preferably a career path where serving in the OT&E 
Authority is seen as career progression by such capable personnel. 

H. CONCLUSION 

The USN OT&E system has. by necessity, developed into a well organised, 
well documented and effective, if complex, system. The RAN system, on the other 
hand, suffers from confusing T&E policy documentation, a weak OT&E 
organisation structure and a general lack of OT&E knowledge and appreciation. The 
determination of OT&E required has traditionally not been achieved early enough 
in the life of a project, and so project funding has not included the provision for 
OT&E. The RAN OT&E system, although originally based on USN OT&E 
philosophy, now differs in the importance, interpretation, application and focus of 
OT&E. As a result of organisation changes, it now lacks the ability to effectively 
manage overall T&E, let alone OT&E, at a time when the operational risk of ship 
projects is increasing. To achieve efficient and effective trials and acceptance of the 
new higher risk warships currently under construction, the RAN OT&E system 
needs to be revised. 

I. A RECOMMENDED MODEL FOR OT&E IN THE RAN 

The OT&E model recommended for the RAN uses principles derived from the 
USN system. By taking consideration of resource limitations, and the characteristics 
and culture of the RAN, it establishes an effective OT&E organisation which is 
important to the testing, evaluation and subsequent acceptance decision of the new 
combatants. 
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1. T&E Definitions 



The basis for this OT&E model are clear definitions of the types and phases 
of T&E. The current Australian definitions are amended and phases of T&E are 
refined to better suit the RAN environment. The following definitions of types and 
phases of T&E are proposed for the RAN system: 

a . Developmental Test and Evaluation (DT&E) 

The DT&E definition remains unchanged as follows: 

DT&E is conducted to assist the engineering design and development process, 
and to verify attainment of technical performance specifications and 
objectives. 

DT&E consists of the three phases listed in Table 2. 



TABLE 2: DT&E PHASES 



DT&E 


Proposed RAN Description 


DT-1 


Validation of design concept. 


DT-2 


Demonstration that design meets specifications. 


DT-3 


Demonstration that production meets required technical 
characteristics or establish standards for first of class. 
PAT&E is a form of DT-3. 



b. Production Acceptance Test and Evaluation (PAT&E) 

The definition of PAT&E is revised to recognise that it is conducted not 

only during the contract period, but also during subsequent RAN testing: 

PAT&E is conducted on production items to ensure systems meet technical 
specifications and requirements, and is a type of DT&E. 
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FAT&E consists of the seven phases listed in Table 3. 



TABLE 3: PAT&E PHASES 



PAT&E 


Proposed RAN Description 


PAT-0 


Design and engineering development tests 


PAT-1 


Production and bum-in tests 


PAT-2 


Environmental qualification tests 


PAT-3 


System development tests 


PAT-4 


Harbour testing 


PAT-5 


Sea testing 


PAT-6 


Certification and Qualification Trials 



c. Operational T est and Evaluation (OT &E) 

The definition of OT&E is revised along USN lines, so that it more 

accurately describes the purpose and nature of OT&E. 

OT&E is conducted to determine a system's operational effectiveness and 
operational suitability, identify system deficiencies and the need for potential 
modifications to meet established OT thresholds, and develop tactics. OT&E 
has three distinguishing characteristics: 

• It is conducted in an operationally representative environment. 

• It is conducted on production representative equipment using fleet personnel 
for operation and maintenance. 

• It is conducted against a threat-representative simulated enemy carrying out 
threat tactics per the latest threat assessment. 
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OT&E consists of the five phases listed in Table 4. 



TABLE 4: OT&E PHASES 



OT&E 


Proposed RAN Description 


OT-1 


Operational assessment of the development proposal to support Full 
Scale Engineering Development approval. 


OT-2 


Demonstration of achievement of program requirements for operational 
effectiveness and suitability to support a production or purchase decision. 


OT-3 


Demonstration of achievement of program requirements for operational 
effectiveness and operational suitability on production of ship/aircraft/ 
system to support Acceptance into Naval Service. Includes limited 
reliability, maintainability, availability and logistic supportability 
assessments. 


OT-4 


Demonstration of achievement of program requirements for operational 
effectiveness and operational suitability on production of ship/aircraft/ 
system in a multi-force, multi-threat environment to support Operational 
Acceptance. Includes detailed reliability, maintainability, availability and 
logistic supportability assessments. 


OT-5 


In-service OT&E, which could include new applications, new tactics, 
revised threat, etc. 



2. T&E in the Acquisition Process 

To ensure the progressive assessment of operational effectiveness and 
operational suitability during the acquisition process, OT&E is scheduled towards 
the end of each acquisition phase. The proposed T&E schedule, noting the 
distinction between DT&E/PAT&E and OT&E, is shown in Figure 1. No change is 
suggested to the current relationship between OT&E phases and acquisition 
milestones, however, the figure separates DT&E/PAT&E from OT&E to clearly 
differentiate these types of testing. 
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Figure 1: T&E in the Acquisition Process 



a . Total Ship Test Program 

The Total Ship Test Program (TSTP) concept, modified from the U.S. 
system, is now well established in RAN shipbuilding projects. The proposed T&E 
phases can be incorporated into an amended TSTP. However, instead of being 
grouped together as in the current TSTP, the DT&E and OT&E events are now 
shown as separate categories of testing. Because these separate DT&E, PAT&E and 
OT&E events are integrated within the overall TSTP, these trials are termed 
collectively as “Integrated Tests and Trials”, which is an extension of the current 
RAN application of this term. Table 5 shows the proposed revised categories of ITT. 
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TABLE 5: CATEGORIES OF ITT 



ITT Part 


ITT Category 


T&E Type 


Description 


ITT Part 1 


ITT-0* 


PAT-0 


Design and engineering development tests 




ITT-1 


PAT-1 


Production and bum-in tests 




ITT-2 * 


PAT-2 


Environmental qualification tests 




ITT-3 * 


PAT-3 


System development tests 




ITT -4 


PAT-4 


Harbour testing 




ITT-5 


PAT-5 


Sea testing 


ITT Part 2 


ITT -6 


DT-3 / PAT-6 


Certification and Qualification Trials 




ITT -7 * 


OT-3 


Operational T&E 


ITT Part 3 


ITT -8 * 


OT4 


Follow on Operational T&E 



Note: * Only performed on first of class 



3. OT&E Organisation 

The current OT&E organisation structure is modified to enable effective 
OT&E to support the acquisition process. It provides for authorities responsible for 
the management, policy making and oversight, specification and conduct of OT&E. 
This proposed reorganisation closely achieves the aim of DT&E/OT&E separation, 
and is shown in Figure 2. 



21 




Figure 2: Proposed RAN OT&E Organisation 



Ideally, the head policy maker and overseer of OT&E should be within 
HQADF or DoD, along the lines of the U.S. system. A shorter term, RAN-only 
solution would be to include this function within the revised Director General Naval 
Policy and Warfare (DGNPW) organisation under the Deputy Chief of Naval Staff 
(DCNS). The Sponsor of major naval acquisitions is the Director General Force 
Development (Sea) (DGFD(Sea)) within HQADF. To be able to determine the 
appropriate nature and extent of OT&E required, the Sponsor needs access to OT&E 
knowledge and experience. A close relationship needs to be maintained by 
DGFD(Sea) with the OT&E policy maker and the OT&E authority (Note dotted 
line). To assist all RAN projects with T&E matters, an authority is required within 
the Materiel Division, under the Assistant Chief of Naval Staff - Materiel) ACMAT- 
N), whose responsibilities include assisting Project Directors in complying with 
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policies, incorporating lessons learned from previous projects and reviewing the 
T&E aspects of projects at major milestones. Although primarily a DT&E policy 
authority, this office would also have an input into the OT&E process by reviewing 
the TEMP, and by advising projects how to prepare for OT&E. The Director General 
Naval Engineering Requirements (DGNER) is currently responsible for the design, 
development, acceptance into service and through life support of ships. This 
engineering policy directorate appears to be well suited to the DT&E policy and 
review function. 

The current OT&E Tester and Evaluator, CTE, within the Maritime 
Command is multifunctional, performing both DT&E/PAT&E and OT&E. To 
achieve the advantages of DT&E and OT&E separation, a revised organisation is 
proposed. All CTE’s current DT&E/PAT&E responsibilities, and those already the 
responsibility of CSO(E), would remain with an in-service trials team under 
CSO(E). CTE’s OT&E duties would be removed and managed by a separate group. 
Commander, OT&E (COT&E), working under an operational authority. As OT&E 
is the assessment of operational effectiveness and suitability, COT&E should be 
responsible to COMFLOT. Headed by a Warfare branch officer, COT&E would 
manage the post delivery trials period for all ships from the MHQ perspective. He 
would still manage the SQT, which, although not true OT&E, still has some 
operational input which could not be met by the in-service trials team under 
CSO(E). 

This proposed reorganisation closely achieves the aim of DT&E/OT&E 
separation with some concessions necessary to limit manpower requirements. Also, 
by including OT&E managed by a Warfare Officer under COMFLOT, OT&E could 
achieve visibility, credibility and influence within the RAN. 
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4. RAN OT&E Policy 



cl Policy Documentation 

To correct the current inconsistent and out of date T&E documentation, 
a major rewrite, rather than just updating is required. This rewrite could be the 
vehicle by which the T&E system within the RAN is overhauled, as before the 
documentation is revised, T&E definitions and terminology need to agreed within 
the RAN and DoD. Then the OT&E and DT&E organisations within the RAN need 
to be resolved, and the appropriate policies decided. 

b. Independence of OT &E Agency 

Given the proposed COT&E is organisationally independent from the 
developing and production agencies, his procedural independence should also be 
maintained to ensure impartiality and validity of testing. When OT&E is conducted, 
it should not be influenced by the Project or Design Approval Authority (DAA). No 
Project or DAA personnel should be present during testing. A final report only 
should be issued by the Maritime Commander, with no progress or interim report 
(unless the testing is unusually lengthy), to allow full analysis and evaluation of the 
test results. Similarly, to ensure test validity and impartiality, contractors should not 
be present during OT&E. 

5. T&E Planning 

Test and Evaluation Master Plans (TEMPs) would be approved early to 
enable inclusion of post delivery PAT&E/DT&E and OT-3 in the Project budget, and 
OT-4 in the Maritime Commander’s budget. To be truly objective, each TEPG 
would be chaired by the authority responsible for ensuring the adequacy of the 
RAN’s overall T&E program. The proposed RAN OT&E policy maker and 
overseer, DGNPW, may best be suited to the chairmanship of each TEPG. 
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6. Land Based Test Sites 

OT&E would be conducted during a dedicated period in the LBTS 
following successful DT&E, to gain an estimate of the potential operational 
effectiveness and suitability of the system. This period would be planned early so as 
to be included in Project schedules and funds. 

7. Whole Ship Testing 

To assess the operational effectiveness and suitability of a ship against the 
user requirement with the degree of confidence required for the Acceptance Board 
to support an AINS decision, a dedicated, free play, scenario based, OT-3 period is 
proposed. This OT-3 would be conducted after the ship has completed the SQT, 
Workup and Operational Readiness Evaluation to ensure that the testing is 
conducted on a worked up and materially proven ship. OT-3 would be conducted of 
the overall ship as a complete weapons system. 

8. RAN Ship Acceptance 

The AB would be convened for the first of class only, providing the 
specialist expertise to interpret requirements and determine the level of acceptability 
required for AINS. The AB consists of Naval Officers drawn part-time from their 
normal jobs, and so the cost appears to be minimal. The AB, however, has a high 
opportunity cost, which is the value of the next best alternative on which the 
members could be working instead of serving on the Board. Once the first of class 
is accepted the Board’s task is complete. The interpretations and levels of 
acceptability established by the AB then guide the Trials Authority in recommending 
acceptance of each of the remaining ships. With a more comprehensive OT-3 period 
proposed to be conducted, the AB members would more effectively use their 
knowledge and experience in assessing the results of OT&E against the user 
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requirement rather than on workmanship and materiel deficiencies. The AB should 
remain as a board of review. 

9. RAN OT&E Procedures 

(L Assessment of Operational Effectiveness 

With the proposed inclusion of a dedicated OT-3 period, the OT&E 
authority will need to develop tests for a free play, scenario based environment 
specifically designed to assess the COIs against the User Requirements. The 
analysis of these tests may be different from the weapons analysis currently 
performed. 

b. Assessment of Operational Suitability 

The assessment of operational suitability, especially for the RAN’s 
current combatant projects, would focus on interoperability and compatibility as 
these present the highest risks with the integration of proven equipment in a unique 
combination in a ship platform. RM&A data is collected and analysed during OT-3 
to give an indication of performance prior to AINS, and then continued for 12 
months during the OT-4 phase. The data would continue to be collected and 
analysed on an “as required”, rather than on a continuous basis by the Trials 
Authority until the RAN develops its routine in-service RM&A data collection and 
analysis system. The analysis of RM&A data is a specialist task, currently outside 
the experience of most uniformed personnel. To achieve significant and valid 
results, an improved RM&A analysis capability would be established. 

c. OT&E Training 

A formal course such as the USN Operational Test Directors Course 
should be a pre-requisite for key billets within the OT&E authority until the RAN 
develops enough knowledge and expertise to develop its own course. Other 
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authorities also require training in OT&E. The OT&E authority should develop 
acquaint courses for other personnel with a need to understand OT&E. Also to 
increase the awareness of OT&E in the RAN, the OT&E authority would deliver 
presentations regularly to Project Directors Courses, the ILS Management and 
Acquisition Course (ILSMAC), Defence management courses, and other 
appropriate acquisition courses. 

The proposed change of the head of the OT&E authority to a Warfare 
Officer as COT&E would provide a career path within the authority. As the 
credibility of OT&E increases within the warfare community, the knowledge and 
experience gained by warfare officers in this area will be seen to be of more benefit 
in future warfare related positions. The proposed employment of COT&E under 
COMFLOT should also be more career enhancing for warfare personnel than the 
current arrangement under CSO(E). 

The professionalism and knowledge of T&E personnel could be 
enhanced by membership in a related professional organisation. The corporate 
membership of a suitable professional organisation may be worthwhile to the RAN. 

J. RECOMMENDATIONS 

It is recommended that the proposed OT&E model be reviewed, evaluated and 
implemented in the RAN to achieve an effective OT&E system to support the 
introduction of operationally effective and operationally suitable new combatants 
into the RAN. 

1. OT&E Definitions 

It is recommended that: 

a. the current definitions and phases of ship T&E be amended to better suit 
the Australian Defence environment. 
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b. the current TSTP be revised to show DT&E and OT&E events as 
separate categories of testing, and these trials be termed collectively as 
Integrated Tests and T rials (ITT). 

2. OT&E Documentation 

It is recommended that: 

a. the RAN T&E DI(N) and ABR 1921 be revised to achieve 
comprehensive and consistent T&E policy guidance. 

3. OT&E in the Acquisition Process 

It is recommended that: 

a. OT&E be conducted, where possible, prior to the production or 
purchase decision in the acquisition process to minimise risk. 

b. OT-3 be separated from the DT&E /PAT&E post delivery trials. 

c. OT-3 be conducted as a dedicated, free play, scenario based period after 
the completion of SQT and ORE to ensure testing is conducted on a 
worked up and materially proven ship. 

d. OT-3 of the overall ship be conducted as a complete weapons system. 

4. OT&E Organisation 

It is recommended that: 

a. the appointment of a DoD OT&E policy maker and overseer within 
HQADF be investigated. 

b. DGNPW be appointed as the OT&E policy maker and overseer within 
the RAN. 

c. a close working relationship on T&E matters be required between 
DGFD(Sea), DGNPW and the OT&E authority. 
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d. DGNER be appointed as the Materiel Division T&E agency to assist 
Project Directors in complying with policies, incorporate lessons 
learned from previous projects, review the T&E aspects of projects at 
major milestones, and advise projects how to prepare for OT&E. 

e. the current CTE organisation be disestablished. 

f. an '‘In-service Trials” team be established under CSO(E) to conduct 
CTE's current DT&E/PAT&E responsibilities, and those already the 
responsibility of CSO(E). 

g. Commander, OT&E (COT&E) be established as the RAN’s OT&E 
authority under COMFLOT, to manage OT&E and the post delivery 
trials period for all ships from the MHQ perspective. 

5. OT&E Policy 

It is recommended that: 

a. policy be developed to enable the performance in the Australian 
environment of foreign equipment to be evaluated efficiently, 
effectively and consistently. 

b. the TEMP approval process be reviewed to ensure appropriate 
authorities are consulted and approve the TEMP. 

c. the TEPG be chaired by DGNPW as the authority appointed to ensure 
the adequacy of the RAN's overall T&E program. 

d. OT&E not be influenced by the Project or DAA, and that Project or 
DAA personnel not be present during testing. 

e. contractors not be present during OT&E. 

f. TEMPs for non-development projects be approved before production 
approval to enable inclusion of post delivery PAT&E/DT&E and OT-3 
in the Project Budget, and OT-4 in MHQ budget. 
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g. OT&E be conducted during a dedicated period in the LBTS following 
successful DT&E, to gain an estimate of the potential operational 
effectiveness and suitability of the system. 

6. Ship Acceptance 

It is recommended that: 

a. the Acceptance Board remain as a board of review. 

b. the Acceptance Board be convened for the first of class only. 

7. OT&E Procedures 

It is recommended that: 

a. The OT&E authority develop tests for a free play, scenario based 
environment specifically designed to assess the COIs against the user 
requirement. 

b. analysis requirements of these tests be assessed, as they may be 
different from the weapons analysis currently performed. 

c. experienced personnel assist in designing suitability tests and analysing 
results. 

d. interoperability and compatibility be a major focus during the 
assessment of operational suitability, especially for the RAM's current 
combatant projects. 

e. an improved RM&A analysis capability be established. 

f. RM&A data continue to be collected and analysed on an 'as required’ 
basis until the RAN develops its routine in-service RM&A data 
collection and analysis system 

g. OT&E reports be made solely by the OT&E authority. 

h. OT&E phase reports be issued only, with no progress or interim reports, 
to allow full analysis and evaluation of the test results. 
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i. the USN Operational Test Director type course be a prerequisite for key 
billets within the OT&E authority until the RAN develops enough 
knowledge and expertise to develop its own course. 

j. the OT&E authority develop acquaint courses for other personnel with 
a need to understand OT&E. 

k. the OT&E authority deliver presentations regularly to Project Directors 
Courses, the ILS Management and Acquisition Course (ILSMAC), 
Defence management courses, and other appropriate acquisition 
courses, so to increase the awareness of OT&E in the RAN. 

l. a Warfare Officer be appointed to head the OT&E authority. 

m. establishing corporate membership of a suitable professional T&E 
organisation be investigated. 
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I. INTRODUCTION 



A. BACKGROUND 

1. Introduction 

Friday, May 15, 1992 marked the end of an era for Operational Test and 
Evaluation (OT&E) in the Royal Australian Navy (RAN). The RAN Trials and 
Assessing Unit (RANTAU), the RAN’s Operational Test and Evaluation authority, 
ceased to exist as an independent unit and was integrated with Maritime 
Headquarters (MHQ) under the Commander, Test and Evaluation. [Ref. 56]This 
marks a continuing shift in the emphasis placed on OT&E within the RAN. 

2. Field of Study 

The fundamental purpose of Test and Evaluation (T&E) in a system’s 
development and acquisition program is to identify the areas of technical risk to be 
reduced or eliminated. Operational Test and Evaluation is the means to demonstrate 
that a ship meets the specified user requirement before being Accepted into Naval 
Service (AINS). This study analyses USN and RAN Ship OT&E organisation, 
methodology, and procedures, and then compares and contrasts the two systems. 
From this analysis, a model for OT&E in the RAN is proposed. 

3. Importance of Research Effort 

In September 1991, the Chief of Naval Staff (CNS) expressed concern that 
the RAN may not be capable of adequately managing the comprehensive and 
complex procedures necessary to accept the ANZAC frigates and Collins class 
submarines into service [Ref. 62]. With these new classes of surface and sub- surface 
combatants under construction, and the RAN currently reorganising its OT&E 
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organisation, the RAN could benefit from a timely analysis of USN and RAN Ship 
OT&E. The study is designed to assist the RAN to develop an effective and efficient 
OT&E organisation. 

B. OBJECTIVES 

OT&E is conducted to determine a system's operational effectiveness and 
operational suitability, to identify system deficiencies and the need for potential 
modifications to meet established operational test thresholds, and to develop tactics. 
It is usually conducted at the end of each phase of the acquisition process to support 
a major milestone decision. The more significant of these is the OT&E conducted 
on a prototype to support a production or purchase decision, and on the first of class 
to support an Acceptance into Naval Service decision (AINS). OT&E is conducted 
against the current user requirement and is definitive feedback to the program’s 
sponsor that the system has met the user requirement. 

OT&E has a high level of representation within the U.S. Department of Defence 
and the USN. The Director, Operational Test and Evaluation, has direct 
responsibility to the Secretary of Defence for prescribing policies and procedures 
governing the conduct of OT&E in the Department of Defence, and is a member of 
the Defence Acquisition Board. The USN has a dedicated command, the 
Commander Operational Test and Evaluation Force (COMOPTEVFOR) 
responsible to the Chief of Naval Operations (CNO) for Ship OT&E. 

The RAN is currently reorganizing it’s OT&E authority. Prior to May 1992, the 
RAN Trials and Assessing Unit (RANTAU) was responsible to the Naval Warfare 
Branch, as the originator of user requirements, for the planning, conduct and 
reporting of OT&E. However, RANTAU has recently been disbanded and the 
OT&E component transferred to the RAN Maritime Commander - the eventual ship 
“user”. 
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Problems were experienced in the RAN with a perceived lack of high level 
representation and consideration of the OT&E requirements during the early phases 
of programs, with the cost and time effectiveness of OT&E, with the conduct of 
OT&E and Developmental Test and Evaluation (DT&E) concurrently, and with 
differences between acceptance of an item from a contractor (Delivery) and 
Acceptance into Naval Service. 

The RAN is a comparatively small navy with limited resources. With large 
indigenous programs including the new Collins class submarines and ANZAC class 
frigates due to undertake OT&E in the near future, it is imperative that the RAN has 
a cost effective and efficient OT&E organisation. 

This study compares the USN and RAN OT&E systems and highlights 
strengths and weaknesses of each. The methodology and procedures currently used 
by each Navy for the assessment of both operational effectiveness and operational 
suitability are analysed. The lessons learned from OT&E of the most recent USN 
ship program, the DDG-51, and the RAN DDG Modernization and Australian 
Frigate programs are reviewed. Then, using the results of this analysis, the current 
RAN plan for the OT&E of the ANZAC class frigate and Collins class submarine 
are reviewed. The output of the study is a model for the organisation and conduct of 
Ship OT&E in the RAN. 

C. THE RESEARCH QUESTION 

The primary research question posed in this thesis is “What are the fundamental 
characteristics of USN ship OT&E and how may USN experience in OT&E be 
applied to achieve an efficient, effective OT&E organisation for the RAN?” 
However, to address the fundamental characteristics of OT&E, a number of 
subsidiary research questions need to be answered. Some of the possible questions 
follow. How does OT&E support the Defence acquisition process? What are the 
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fundamental characteristics of the USN OT&E organisation? What are the 
fundamental characteristics of USN OT&E policy? What are the fundamental 
characteristics of USN OT&E procedures? How is USN OT&E implemented? How 
does OT&E support the acceptance of ships into the USN Fleet? How do the USN 
and RAN OT&E systems currently compare? What needs to be done in the RAN to 
achieve an effective OT&E system with a minimum of additional resource 
requirements? 

D. SCOPE, LIMITATIONS AND ASSUMPTIONS 

The scope of this thesis is limited to the study of OT&E for naval warships and 
submarines. Although based on the same philosophy, the OT&E of aircraft and 
equipment is not addressed. Although other nations conduct OT&E (e.g.. United 
Kingdom), this study is limited to the analysis of USN and RAN OT&E systems 
only. This study is an overview of OT&E organisation, methodology and 
procedures. It does not address specific implementation matters or details of 
individual tests or trials. This study is limited to UNCLASSIFIED material only, 
and the collection of RAN information was limited to written and verbal 
correspondence as no personal interviews were able to be conducted with RAN 
authorities. No substantial increase of RAN OT&E resources in the near future was 
assumed. 

E. WHY STUDY USN OT&E? 

Study of the USN OT&E system could be beneficial to the RAN for a number 
of reasons. Due to the RAN’s long association with the USN and its purchase of U.S. 
ships and equipment, it has a similar philosophical approach to acquisition and 
T&E. The basis of much of the current RAN T&E policies, methods and procedures 
is derived from the USN. 
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Technology is changing the face of modern warfare, and the USN is at the 
leading edge of this movement, adapting the best of new technologies to proven 
concepts and platforms. The USN strives to ensure that, as technology evolves, it is 
used to best advantage. As OT&E determines a new system’s operational 
effectiveness and operational suitability, the need for potential modifications to 
meet established OT thresholds, and develops tactics, it is a critical part of adapting 
technology to modem warfare. 

Being the most modem and capable naval force in the world today, the USN has 
substantial weapons system acquisition, operational and combat experience. This 
experience has confirmed their need for OT&E, and refined their OT&E 
organisation, policies, procedures and methodology. The RAN could benefit from 
this experience. 

The USN are world leaders in the development of high technology weapons 
systems and have extensive experience in their acquisition, testing and operation. As 
the RAN has a similar philosophical approach to acquisition and T&E, the USN 
OT&E system forms a good basis from which to develop an RAN OT&E 
organisation, policies, procedures and methodology. 

F. RESEARCH METHODOLOGY AND LITERATURE REVIEW 

A balanced research methodology was used following the general research 
guidance outlined in Reference 1. The research consisted of three strategies: 

1. Archival Research 

Archival research consisted of examining the relevant formal documents 
pertaining to OT&E in both navies. This included a review of U.S. Defence 
Acquisition law, followed by review of the U.S. DoD and USN official 
documentation, pertaining to T&E, and to ship OT&E in particular. A similar 
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approach was taken in reviewing the Australian documentation. Also, a literature 
search was conducted to elicit the latest published views and opinions on OT&E. 
This search was conducted through two primary sources: 

a. Database Search 

A search was conducted with DIALOG Information Services, a 
commercial information service available through the Dudley Knox Library, Naval 
Postgraduate School. The DIALOG service is a computer database accessible by 
modem. The search was conducted using a number of key words including 
Operational and Evaluation, policy, procedures, and destroyer and frigate. A listing 
of 200 recent items was provided [Ref. 2]. 

b. Defence Logistics Studies Information Exchange 

The Defence Logistics Studies Information Exchange (DLSIE), located 
at the U.S. Army Logistics Management College, Fort Lee, Virginia, is a U.S. DoD 
organisation that acquires, organises, stores and disseminates logistics and 
management information on a DoD wide basis [Ref. 3], A custom bibliography was 
requested on the subject of ship Operational Test and Evaluation. A listing of 48 
studies and reports on the subject was provided [Ref. 4], 

Other sources of information included journals of professional 
organisations and Defence magazines, however, the most useful guidance to 
relevant publications was provided by personnel involved in OT&E. In summary, 
the archival research established the current, formal organisation, methodology and 
conduct of OT&E in both navies, and identified the latest public opinions on OT&E 
issues. 
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2. Personal Interview Research 

Ship OT&E is a highly specialised, evolving field of expertise. Many 
developments are not published formally by DoD or in the commercially available 
literature. It was necessary, therefore, not only to survey the literature on the subject, 
but also to conduct interviews. This research was conducted by personal interviews, 
either in person or over the phone, with key participants in the ship OT&E field. 
Views and opinions were sought from authorities in both navies who either conduct, 
use the results of, or are otherwise effected by ship OT&E. The list of pertinent 
references as well as personnel interviewed are contained in the References section 
of this thesis. 

3. Research Analysis 

Research analysis was then applied to the information gathered by the other 
two strategies, to assess the information, evaluate the strengths and weaknesses of 
each system, and to develop the OT&E model. This analysis consisted of breaking 
down the USN and RAN OT&E systems into their component parts, in order to 
determine how (and where possible, why) the systems are organised, and particular 
procedures and methods used. An OT&E model for the RAN was then developed 
from the results and conclusions of the research analysis. As the RAN has a similar 
philosophical approach to acquisition and T&E to the USN, their OT&E system 
forms a good basis from which to further develop RAN OT&E organisation, 
policies, procedures and methodology. 

G. DEFINITIONS AND ABBREVIATIONS 

Terms important to the analysis of OT&E systems in the RAN and USN are 
defined as they are introduced. A list of acronyms is included in Appendix B. 



7 



H. ORGANISATION OF THESIS 



The study initially reviews the background of OT&E in Chapter II. The purpose 
of T&E in Defence acquisition is discussed and the types of T&E applicable to ships 
are introduced. The U.S. and Australian strategic, economic and Defence 
environments are discussed briefly and compared. Some similarities and differences 
between the USN and RAN are also noted. The philosophy, organisation and 
management of OT&E within the USN and RAN are then analysed in Chapters III 
and IV respectively. A comparative analysis of USN and RAN OT&E systems is 
included in Chapter V. As result of this analysis, a model for OT&E in the RAN is 
proposed in Chapter VI. Chapter VII details the conclusions and Chapter VIII lists 
the recommendations. 
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II. OPERATIONAL TEST & EVALUATION BACKGROUND 



This chapter presents background information considered necessary for a better 
understanding of OT&E. The purpose of T&E in Defence acquisition is discussed 
and the types of T&E applicable to ships are introduced. The U.S. and Australian 
strategic, economic and defence environments are discussed briefly and compared. 
Some similarities and differences between the USN and RAN are also noted. 

A. PURPOSE OF T&E IN DEFENCE ACQUISITION 

The fundamental purpose of T&E in a defence system’s development and 
acquisition program is to identify the areas of technical risk to be reduced or 
eliminated. During the early phases of development, T&E is conducted to 
demonstrate the feasibility of conceptual approaches, to minimise design risk, to 
identify design alternatives, to compare and analyse trade-offs, and to estimate 
operational effectiveness and suitability. As a system undergoes design and 
development, the emphasis in testing moves gradually from developmental test and 
evaluation, which is concerned chiefly with the attainment of engineering design 
goals, to operational test and evaluation, which focuses on questions of operational 
effectiveness, suitability and supportability. [Ref. 18:p. 1-1] A primary contribution 
made by T&E is the identification and reporting of deficiencies that may adversely 
impact the performance capability, availability or supportability of a system. 

T&E serves a number of useful functions, providing information for the 
following customers: 

• Developers to assist in the identification and resolution of technical 
difficulties. 
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• Decision makers responsible for making the investment decision to procure 
a new system and for deciding on the most effective use of limited resources. 

• Operational users to support the development of effective tactics, doctrine 
and procedures. [Ref. 18:p. 1-1] 

R TYPES OF SHIP TEST AND EVALUATION 

The principal types of T&E pertaining to ships are: 

1. Developmental Test and Evaluation (DT&E) 

DT&E is conducted to assist the engineering design and development 
process and to verify attainment of technical performance specifications and 
objectives [Ref. ll:p. 3]. 

2. Production Acceptance Test and Evaluation (PAT&E) 

PAT&E is testing conducted on production items to ensure systems meet 
contract specifications and requirements, usually for contractual acceptance 
purposes. It is a type of DT&E [Ref. 1 l:p. 5]. 

3. Operational Test and Evaluation (OT&E) 

OT&E is conducted to determine a system’s operational effectiveness and 
operational suitability, identify system deficiencies and the need for potential 
modifications to meet established operational thresholds, and develop tactics [Ref. 
1 l:p. 5A]. OT&E has three distinguishing characteristics: 

• It is conducted in an operationally representative environment. 

• It is conducted on production representative equipment using fleet personnel 
for operation and maintenance. 

• It is conducted against a threat-representative simulated enemy carrying out 
threat tactics per the latest threat assessment. 
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4. The Difference between DT&E and OT&E 



DT&E is focused on a system meeting technical and production 
specifications, while in OT&E, the focus is on assessing the actual functioning of the 
system in the realistic combat environment, against the user requirement. Figure 1 
illustrates this principal difference between DT&E and OT&E. 




Figure 1: DT&E and OT&E Comparison 



DT&E is planned and monitored by the developing agency and is normally 
initiated by the contractor. It includes the T&E of components, sub-systems and 
hardware and software integration, as well as preproduction and production 
qualification testing. OT&E, on the other hand, is conducted by an organisation that 
is independent of the developer, on the complete system in as operationally realistic 
an environment as possible, with hostile forces representative of the anticipated 
threat and with typical users operating and maintaining the system. Table 1 
highlights the differences between DT&E and OT&E. 
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TABLE 1: DIFFERENCES BETWEEN DT&E AND OT&E [Ref. I8:p. 3-5] 



DT&E 


OT&E 


Controlled by Program Manager 


Controlled by Independent Agency 


One-on-One Tests 


Many-on-Many Tests 


Sterile Controlled Environment 


Tactical Environment with Operational 
See nano 


Contractor Involvement 


No Contractor Involvement 


Trained Experienced Operators 


Users Recently Trained on Svstem 
* * 


Specific Performance Measurements 
and Goals 


Operational Effectiveness and Operational 
Suitability Performance Measurements 



To further highlight the difference between DT&E and OT&E, this 
anecdote, originally from the Air Force Manual 55-43, 1979 and reprinted in the 
U.S. Defence Systems Management College Test and Evaluation Guide [Ref. 18:p. 
3-3] is an account of what is probably the first operational test and evaluation: 

The test and evaluation of aircraft started with the contract awarded to the 
Wright brothers in 1908. This contract specified a craft which would lift two 
men with a total weight of 350 pounds, carry enough fuel for a flight of 125 
miles, and fly 40 miles per hour in still air. The contract also required that 
testing be conducted to assure this capability. What we now call DT&E was 
satisfied when the Wright brothers (the developer) demonstrated that their 
airplane could meet those first contract specifications. However, no immediate 
military mission had been conceived for the Wright Flyer. It was shipped to 
Fort Sam Housten, Texas where Captain Benjamin D. Foulois, the pilot, had 
orders to ‘'teach himself to fly”. He had to determine the airplane's 
performance, how to maintain it, and the kind of organisation that would use 
it. In the process. Captain Foulois subjected the Wright Flyer to test and 
evaluation in operational conditions. Foulois soon discovered operational 
deficiencies. For example, there was no seat on the airplane. During hard 
landings. Foulois’ 130 pound frame usually parted company from the airplane. 
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To correct the problem, Foulois bolted an iron tractor seat to the airplane. The 
seat helped but Foulois still toppled from his perch on occasion. As a further 
improvement, Foulois looped his Sam Browne belt through the seat and 
strapped himself in. Ever since then, contoured seats and safety belts - a 
product of this earliest ‘"operational” test and evaluation - have been part of the 
military airplane. 

5. Operational Effectiveness and Operational Suitability 

A system’s operational effectiveness and operational suitability are the 
fundamental objects of OT&E. The USN and RAN definitions (which are possibly 
derived from the USN) for these terms are similar. 

cl Operational Effectiveness 

(1) USN Definition. Operational effectiveness is the capability of the 
system to perform its intended function effectively over the expected range of 
operational circumstances, in the expected environment, and in the face of the 
expected threat, including countermeasures where appropriate [Ref. 1 l:p. 8J. 

(2) RAN Definition. Operational Effectiveness is the capability of the 
system to perform its intended function to the required standard, over the 
expected range of operational circumstances, in the expected environment, and in 
the face of the expected threat including countermeasures[Ref. 44:p. A-l]. 

A subtle difference between these definitions is in the description of the 
assessed standard. The USN uses the term “effectively”, meaning to do the job 
regardless of meeting specification, whereas the RAN uses the term “to the required 
standard”, meaning to meet the specification. 

b. Operational Suitability 

(1) USN Definition. Operational suitability is the capability of the 
system, when operated and maintained by typical fleet personnel in the expected 
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numbers and of the expected experience level, to be reliable, maintainable, 
operationally available, logistically supportable when deployed, compatible, 
interoperable, and safe[Ref. 1 l:p. 9], 

(2) RAN Definition. Operational suitability is the capability of the 
system, when operated and maintained by operational personnel in the expected 
numbers and of the expected experience level, to be reliable, maintainable, 
operationally available and logistically supportable [Ref. 44:p. A-2] in the specified 
environment within a specified time period. [Ref. 38:p. 14A-3] 

The relationship of operational effectiveness and operational suitability 
to OT&E, and the major questions addressed in OT&E are shown in Figure 2. 




Figure 2: Questions Addressed in OT&E [Ref. 18:p. 3-6] 
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C. AUSTRALIA’S DEFENCE ENVIRONMENT 



The recently publicly released “Australia's Strategic Planning in the 1990’s’’ 
[Ref. 5] has not identified a specific military threat to Australia in the foreseeable 
future. It does, however, identify significant uncertainties concerning the future 
shape of the Australian strategic environment. 

1. Strategy 

Australia’s national strategic policy has evolved over recent decades. It has 
come from a position of defence dependence on allies through concentration on the 
immediate needs of self defence to a positive acceptance of both self-reliance and 
regional influence [Ref. 5:p. 3]. The latest Australian Defence Corporate Plan [Ref. 
8] highlights three central elements in Australia’s defence policy: 

• Self Reliance 

• Regional Cooperation 

• Strong Alliances 

Self reliance accords priorities to meeting credible levels of threat in 
Australia’s area of direct military interest [Ref. 6:p. vii]. Regional cooperation 
ensures strategic stability and security in the region, while strong alliances express 
common security interests and provide avenues for defence access and strategic 
influence. Close defence relations with the U.S. remain central to the policy of 
defence self-reliance. Although Australia does not depend on the U.S. for protection, 
it derives significant benefits from close collaboration through access to advanced 
technology, training and exercises, and intelligence. These benefits give Australia a 
level of capability that would be difficult and more costly for it to achieve alone. 
[Ref. 8:p. p 5] 
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2. Australian Defence Budget and Policies 

To achieve its defence goals, Australia’s defence budget for 1992-93 is $A 
9.8 billion. This accounts for about 9% of Federal outlays and is equivalent to 2.4% 
of the expected GDP. [Ref. 8:p. 23] Australia is placing emphasis on capital 
investment, devoting 27.3% of its defence budget. It intends to maintain capital 
investment at about $A 2.5 billion per year, of which about $A 2 billion is on capital 
equipment. Along with this significant investment in new capabilities, Australia is 
now buying far less “off the shelf’ from overseas, with systems increasingly being 
designed and built in Australia, or adapted from overseas designs. [Ref. 8:p. 17] As 
a result, Australian is now building frigates and submarines due to enter service in 
the mid 1990's. 

The Australian Government’s forward estimates indicate that the Defence 
Budget will be reduced by 0.5% in 1993-94 and then maintained at this level in real 
terms in 1994-95 and 1995-% [Ref. 8:p. 24]. Over the four years to 19%, reductions 
of up to 6,600 permanent military personnel (about 10% of total force) and 1785 
Defence civilians (about 8%) are anticipated [Ref. 8:p. 14]. 

The area over which the Australian Defence Force (ADF) needs to operate, 
the importance of denying an adversary freedom in the sea and air approaches, 
Australia’s limited population base and the characteristics of the northern Australian 
area, place a premium on technologically advanced systems rather than manpower 
intensive forces. As the costs of local development and production of 
technologically advanced systems are high, development emphasis is placed on 
those areas unique to Australia or where particular skills have ongoing relevance to 
a range of defence capabilities.[Ref. 5:pp. 28, 38] 

In summary, Australia is contributing to its strategic goals and ensuring that 
the ADF remains a formidable military force in regional terms, by maintaining 
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priority on capital investment. The focus is towards a smaller and more technology 
oriented force, with a more streamlined and effective organisation. Therefore, 
Australia needs to ensure that its more technology advanced systems are 
operationally effective and operationally suitable for the missions required of them. 

3. U.S. Versus Australia 

Of a similar physical size but much larger population, the U.S. is an 
economic and military superpower compared to Australia. Table 2 compares some 
significant characteristics of the two nations. 



TABLE 2: U.S. VS. AUSTRALIA 1991/92 [Ref. 10] 



1991 Statistics 


U.S. 


Australia 


Area 


3,021,295 sq m 


2,966,151 sq m 


Population 


251.8 million 


17.1 million 


GDP 


$US 5,673.9 billion 


$A 376.3 billion 
($US 293.2 billion) 


Growth 


-0.8% 


-2.2% 


Inflation 


4.3% 


3.3% 


Debt 


$US 828.5 billion 


$US 125.0 billion 


Defence Budget 


$US 290.9 billion 


$A 9.44 billion 
($US 7.27 billion) 


% of Federal Outlay 


22% 


9% 


Total Armed Forces 


1,913,750 


67,900 


Spending on Capital 


$US 71.7 billion 


$A 1.92 billion 


Equipment 




($US 1.40 billion) 


% of Defence Budget 


26.4% 


20.4% 


R & D Expenditure 


$US 36.2 billion 


$A 0.22 billion 
($US 0.16 billion) 


% of Defence Budget 


12.4% 


2.3% 
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a. Differences 



The “National Military Strategy of the United States” [Ref. 7] affirms 
that U.S. Defence policy is built upon four key foundations: 

(1) Strategic Deterrence and Defence. The significant political 
instabilities in the former Soviet Union and the threat posed by the increasing 
numbers of potentially hostile states developing weapons of mass destruction 
requires that the maintenance of a modern, fully capable and reliable strategic 
deterrent remains the first defence priority. 

(2) Forward Presence. The day to day presence of U.S. forces in 
regions vital to U.S. national interests is key to averting crises and preventing war. 
In addition to forces based overseas and afloat, forward presence includes periodic 
and rotational deployments, access and storage agreements, combined exercises, 
security and humanitarian assistance, port visits and military to military contacts. 

(3) Crisis Response. The capability to respond to regional crises on 
short notice is a key demand of U.S. strategy. This response may range from a single 
discriminate strike to the employment of overwhelming force to defeat a regional 
aggressor. 

(4) Force Reconstitution. Reconstitution is required to preserve a 
credible capability to forestall any potential adversary from competing militarily 
with the U.S. It involves forming, training and fielding new fighting units if required 
at short notice. It also involves maintaining technology, doctrine, training, 
experienced military personnel and innovation necessary to retain the competitive 
edge in decisive areas of potential military competition. 

This strategy is founded on the premise that the United States will 
provide the leadership needed to promote global peace and security [Ref. 7:p. 6]. It 
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does, however, reflect the shift from containing the spread of communism and 
deterring Soviet aggression to a more diverse, flexible strategy which is regionally 
oriented and capable of responding decisively to future challenges [Ref. 7:p. 1], 
Although the U.S. and Australian strategic policy elements may not be directly 
comparable, the U.S. places a greater emphasis on Defence with strategies of world 
leadership, global influence and self sufficiency, while Australia places emphasis on 
regional influence and self-reliance. 

To achieve these goals, the U.S. devotes a considerably higher 
proportion of its Federal expenditure to Defence, than does Australia. It also spends 
a considerably greater percentage of its Defence Budget on R&D. U.S. policy is to 
maintain world leadership in key Defence technologies, by continuously improving 
the process of indentifying and introducing new technologies [Ref. 9]. Although 
R&D is conducted with emphasis on those areas unique to Australia, lower risk, 
proven technologies are preferred for the majority of new Australian Defence 
acquisitions. 

b. Similarities 

Like Australia, the U.S. current projections for FY 92-95 show a 
continued reduction of the Defence Budget by about 2% per annum. Additionally, 
with the recent change to a Democratic Administration under President Clinton, 
further reductions in Defence expenditure can be expected. During this period of 
budget and trade deficits and urgent domestic needs, the U.S. military strategy will 
be implemented within a significantly reduced Defence budget, and so places a 
premium on efficiency without compromising effectiveness [Ref. 7:p. 4]. 

As does Australia, the U.S. relies on technological superiority to offset 
quantitative disadvantages, to minimise risk to its forces, and to enhance deterrence 
and the potential for swift, decisive termination of conflict [Ref. 7:p. 10]. The U.S 
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also realises it can meet the challenges of the foreseeable future with a much smaller 
force than it has had in recent years [Ref. 7], and that implications of a 25% force 
cut by 1995 may be possible [Ref. 10]. Despite these funding and program cutbacks, 
the U.S. is also building new surface combatants and submarines. The first DDG-51 
“Arleigh Burke" completed testing and the first SSN-21 “Seawolf’ is under 
construction. 

A significant similarity between the U.S. and Australia is one of 
culture. Both nations have predominantly Caucasian, English speaking populations, 
a democratic system of government, close economic, military and political ties, and 
share similar cultural values and ideology. 

Despite being significantly larger than Australia in economic and 
military terms, and, consequently, having a different Defence strategy, the U.S. is 
similar to Australia in many areas. These include reducing budgets and size of the 
Defence force, a requirement for advanced technology, and a need for efficiency 
while maintaining effectiveness. But perhaps above all, a similar culture which 
promotes a relative ease of communication and assimilation of ideas, concepts, 
procedures and methodologies between the two nations. 

4. United States Navy Versus Royal Australian Navy 

A similar contrast applies to the navies of both nations. As Table 3 shows, 
the USN is far larger and more capable than the RAN. However, there are number 
of similarities. Although the RAN had its origins in the Royal Navy and with British 
systems, since the 1960’s the greater influence on the RAN has been from the USN. 
The purchase of three modified Charles F. Adams (DDG-2) class destroyers from the 
U.S. in the mid 1960’s heralded an increasingly close relationship between the two 
navies. The purchase of four Oliver Hazard Perry (FFG-7) class frigates from the 
U.S. in the early 1980’s, and the local construction of two more of the class in 
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Australia, continued this relationship. Although the RAN’s needs no longer support 
the purchase of current U.S. ship types, many new systems and equipment are still 
purchased through the USN. Ongoing support for these U.S. ships currently in RAN 
service, and Australia’s emphasis on interoperability with allies, ensure a continuing 
close association. The RAN shares with the USN much common equipment and 
weapons, compatible operational procedures, similar logistic support systems and 
defence acquisition procedures. 



TABLE 3: USN VS. RAN 1991 [Ref. 10] 





USN 


RAN 


Personnel 


546,650 


15,300 


Submarines 


110 


5 


Aircraft Carriers 


14 


- 


Cruisers 


48 


- 


Destroyers 


45 


3 


Frigates 


83 


8 


Patrol (inc. Coast Guard) 


30 


15 


Mine Warfare 


24 


2 


Amphibious 


65 


1 


Support & Misc. 


162 


12 


Total Ships: 


581 


46 


Major Ship 
Acquisition Programs 


DDG-51 class destroyers 
SSN-21 class submarines 


ANZAC class frigates 
COLLINS class subs. 



D. OT&E BACKGROUND SUMMARY 

OT&E forms a fundamental part of minimising risk in the weapons system 
acquisition process and in future system operations. It is conceptually different from 
DT&E, being focussed on assessing the operational effectiveness and operational 
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suitability of the system in a realistic combat environment, with respect to the user 
requirement. 

Australia is placing emphasis on capital investment for Defence, particularly for 
the RAN, with new frigates and submarines presently under construction. In 
response to budget reductions the current focus of the RAN is towards a smaller and 
more technology oriented force, with a more streamlined and effective organisation. 

The U.S. and Australia have a number of similarities, perhaps the most 
important, a similar culture which promotes a relative ease of communication and 
assimilation of ideas, concepts, procedures and methodologies between the two 
nations. The USN is a world leader in the development of high technology weapons 
systems and has extensive experience in their acquisition, testing and operation. 

With new frigates and destroyers due to enter service in the mid 1990s, the RAN 
has an urgent need for an efficient, effective OT&E system to ensure the operational 
risk of these new combatants is minimised, and the combatants meet the current 
requirements of the RAN. 






III. USN SHIP OPERATIONAL TEST AND EVALUATION 



This chapter describes the USN OT&E system. It explains the OT&E 
relationship and importance to the U.S. Department of Defence (DoD) acquisition 
process, the specific OT&E policies and methods employed in ship OT&E and why 
the OT&E of ships is different from other systems. The USN system of ship 
acceptance and the input of OT&E into the process is also described. The OT&E of 
the DDG-51 class destroyer, the latest class of warship to undergo OT&E, is used as 
an example of the implementation of USN OT&E policies. The workings of the 
USN system are well documented and the OT&E system can be traced from the 
statute requirements of Congress, through DoD directives to implementation by 
Navy specific instructions. 

A. U.S. DEPARTMENT OF DEFENCE ACQUISITION PROCESS 

The USN OT&E system forms an integral part of the US DoD acquisition 
process. In 1987, the Defence system acquisition process underwent revision in an 
attempt to make it less costly, less time consuming, and more responsive to the needs 
of the operational community. The Defence system life cycle consists of five phases 
of the acquisition process: 

• Concept Exploration/Definition 

• Demonstration/Validation 

• Engineering and Manufacturing Development 

• Full Rate Production/Deployment 

• Operational Support 
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As Figure 3 shows, these phases are separated by major decision points 
(milestones) when a program is reviewed and authorised to advance to the next stage 
in the cycle, that provide a basis for progressive decisionmaking associated with 
program maturation. T&E results and planned T&E play an important part and are 
rigorously assessed as part of the milestone reviews [Ref. 18:p. 1-3]. These 
milestones are: 

• Milestone O - Concept Studies Approval 

• Milestone 1 - Concept Demonstration Approval 

• Milestone II - Development Approval 

• Milestone IIA - Low Rate Initial Production 

• Milestone III - Full Rate Production 

• Milestone IV - Major Modification Approval (as required) 
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Figure 3: U.S. Acquisition Process for Major Programs 
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B. TYPES OF USN SHIP TEST AND EVALUATION 



The USN Instruction on Test and Evaluation [Ref. 11] defines the principal 
types of T&E pertaining to ships as follows: 

1. Developmental Test and Evaluation (DT&E) 

DT&E is conducted to assist the engineering design and development 
process and to verify attainment of technical performance specifications and 
objectives [Ref. 1 l:p. 3]. DT&E is conducted in three major phases. 

2. Production Acceptance Test and Evaluation (PAT&E) 

PAT&E is testing conducted on production items to ensure systems meet 
contract specifications and requirements, and is a type of DT&E [Ref. 1 l:p. 5]. The 
USN regard trials of new ships conducted by the Board of Inspection and Survey 
(INSURV) and shipyard industrial testing at PAT&E [Ref. 32:p. 2-5]. Two phases of 
ship PAT&E are defined: 

• Ship Construction Tests and Trials consist of all testing conducted on the ship 
during construction, including INSURV’s Acceptance Trials. 

• Ship Post-Delivery Tests and Trials are the conventional tests and trials, 
including INSURV’s Final Contract Trials, that commence after ship 
delivery and continue to the end of the SCN obligation or work limiting date. 

3. Operational Test and Evaluation (OT&E) 

OT&E is conducted to determine a system’s operational effectiveness and 
operational suitability, identify system deficiencies and the need for potential 
modifications to meet established OT thresholds, and develop tactics [Ref. ll:p. 
5A]. OT&E has three distinguishing characteristics: 

• It is conducted in an operationally representative environment. 

• It is conducted on production representative equipment using fleet personnel 
for operation and maintenance. 
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• It is conducted against a threat-representative simulated enemy carrying out 
threat tactics per the latest threat assessment. 

OT&E is subdivided into two major categories: 

• Initial OT&E (IOT&E) (OT-I and OT-II) which is up to and including 
OPEVAL (final phase of OT-2). 

• Follow-on OT&E (FOT&E) (OT-III and OT-IV) which is all OT&E after the 
final phase of OPEVAL. 

C. OT&E IN THE ACQUISITION PROCESS 

According to the Honorable Gerald A. Cann, Assistant Secretary of the Navy 
for Research. Development and Acquisition: 

Test and evaluation is the major control mechanism in the acquisition process 
for assessing system performance against system requirements. Navy decision 
makers rely on solid testing results, both developmental and operational, to 
provide the analytical groundwork for forming judicious program 
decisions. [Ref. 12] 

This statement by the USN’s senior acquisition executive, highlights the 
importance of T&E within the USN. Program advance from one phase to the next is 
not by the calendar of planned schedule, but by actual resolution of critical 
operational issues and achievement of pre-set thresholds verified by T&E [Ref. 
1 l:p. 2]. The importance of OT&E, in particular, to the progressive assessment of 
programs is such that its use has been mandated by the U.S. Congress and 
incorporated into the laws of the United States. 

1, Statute Requirement 

The requirement for adequate OT&E of Defence programs has strong 
congressional support and is mandated by law. The law includes the provision of a 
Director of Operational Test and Evaluation in the Department of Defence, 
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appointed from civilian life by the President, by and with the advice and consent of 
the Senate [Ref. 16]. The law also addresses specific areas of OT&E reporting and 
conduct to ensure the Congress is kept informed, and the testing and reporting are 
impartial [Ref. 17]. By including the requirement that a major Defence acquisition 
program may not proceed beyond low-rate initial production (i.e. for Milestone III 
- Full Rate Production Decision) until Initial OT&E of the program is completed. 
Congress ensures that OT&E is an integral part of the acquisition process. 

2. OT&E Contributions at Major Milestones 

T&E progress is monitored by the Office of the Secretary of Defence 
(OSD) throughout the acquisition process. For major Defence acquisition programs, 
the Director, T&E and the Director, OT&E within OSD render independent 
assessments to the Defence Acquisition Board, the Defence Acquisition Executive 
and Secretary of Defence at each major milestone. These T&E officials also assess 
the T&E results for less than major defence acquisition programs that are 
specifically designated by OSD as OSD T&E Oversight Programs. The assessments 
are based on the following T&E information: [Ref. 18:p. 2-5] 

• The Test & Evaluation Master Plan (TEMP) and more detailed supporting 
documents developed by responsible Service activities. 

• Service operational test agency reports and briefings. 

• Developmental test and evaluation data. 

a. OT &E prior to Milestone I 

During the Concept Exploration / Definition Phase prior to Milestone I, 
laboratory testing, modeling and simulations are conducted to demonstrate and 
assess the capabilities of key subsystems and components. The test and simulation 
designs are based on the requirements documented in the Mission Need Statement 
(MNS). The Operational Test and Evaluation Agency (OTA) monitors concept 



27 



exploration T&E to gather information for future test and evaluation planning and 
to provide effectiveness and suitability inputs desired by the Program Manager. The 
OTA also conducts operational assessments, as feasible, to assess the operational 
impact of candidate technical approaches and to assist in selecting preferred 
alternative systems concepts. [Ref. 18:p. 2-5] 

b. OT &E prior to Milestone II 

During the Demonstration/Validation phase prior to Milestone II, 
concepts approved for demonstration and validation form the baseline that is used 
for detailed test planning. The OTA conducts early operational assessments to 
identify the best approach, indicate the risks and solutions for this phase of 
development, examine operational aspects of the systems development, and 
estimate potential operational effectiveness and suitability. This OT&E phase is an 
assessment only, with no actual testing being conducted. Typical operational and 
support personnel are used to obtain a valid estimate of the user's capability to 
operate and maintain the system. The user of the system monitors test and evaluation 
during the concept Demonstration and Validation phase. Among the most important 
products of user monitoring are the attainment of early orientation and training, 
demonstrations of system performance and valid operational test assessments of 
systems maintainability and supportability.[Ref. 18:p. 2-7] 

c. OT &E prior to Milestone III 

The objective of the Engineering and Manufacturing Development 
(EMD) phase prior to Milestone III, is to design, fabricate and test a preproduction 
system that closely approximates the final product. T&E activities intensify during 
EMD, culminating in OPEVAL, and make significant contributions to the overall 
acquisition decision process. [Ref. 18:p. 2-9] OT&E conducted prior to the 
production decision is for the following reasons: 
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• Estimate the operational effectiveness and suitability of the system. 

• Identify operational deficiencies. 

• Recommend and evaluate changes in production configuration. 

• Provide information for developing and refining logistics support 
requirements for the system and training, tactics, techniques, and doctrine. 

• Provide information to refine operation and support cost estimates, and to 
identify system characteristics or deficiencies that can significantly impact 
these costs. 

• Determine whether the technical publications and support equipment are 
adequate. 

• Estimate the survivability of the system in the operational environment. 

d. OT&E after the Production Decision 

OT&E activities continue after the production decision in the form of 
Follow-on Operational Test and Evaluation (FOT&E) consisting of OT-III and OT- 
IV. It is accomplished to verify the operational effectiveness and suitability of the 
production system and to determine if deficiencies identified during initial OT&E 
(OT-III) have been corrected. A second phase of FOT&E (OT-IV) may be conduced 
by the user to refine doctrine, tactics, techniques and training programs over the life 
of the system. [Ref. 18:p. 2-10] 

D. US OT&E ORGANISATION 

Major participants in the US OT&E process are listed in Table 4. The US OT&E 
community consists of a mix of service and OSD offices responsible for planning, 
programming, budgeting and evaluating operational tests. Figure 4 gives an outline 
of the OT&E organisation. 
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TABLE 4: USN OT&E FUNCTIONAL AUTHORITIES 



OT&E Function 


USN Authority 


Policy Maker & Ov erseer 


Director, Operational Test & Evaluation (DoD) 


Sponsor 


Chief of Naval Operations 


Developing Agency 


Commander, Naval Sea Systems Command 


Operational Tester & Evaluator 


Commander, Operational Test & Evaluation Force 


User 


Fleet Commanders, Ships’ Commanding Officers 


Coordinator 


Director, T&E and Technology Requirements 



The US OT&E community consists of a mix of service and OSD offices 
responsible for planning, programming, budgeting and evaluating operational tests. 
Figure 4 gives an outline of the OT&E organisation. 




Figure 4: U.S. OT&E Organisation 
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1. Policy Maker and Overseer 

The Director of OT& E (DOT&E) is the principal OT&E official in the 
Department of Defence and the principal advisor to the Secretary of Defence on 
OT&E. He is responsible for policy formulation, evaluation and oversight of OT&E 
within DoD. With status comparable to an Assistant Secretary of Defence, DOT&E 
is independent of other DoD officials and reports directly to the Secretary of 
Defence and additionally to Congress. His responsibilities include the following 
[Ref. 13:p. 7-4]: 

• Monitoring and reviewing all OT&E within DoD. 

• Designating observers to be present during preparation for and conduct of the 
testing portion of OT&E. 

• Controlling joint OT&E and coordinating OT&E conducted by more than 
one Military Department or Defence Agency. 

• Analysing the results of major system acquisition OT&E. For major systems 
and DOT&E oversight programs, reporting to SECDEF and to Congressional 
Armed Services and Appropriations Committees that OT&E is adequate and 
confirms effectiveness and suitability for combat systems tested. 

• Making recommendations to SECDEF on all budgetary and financial matters 
pertaining to OT&E, including facilities and equipment. 

• Approving OT&E plans for major Defence acquisition programs and 
DOT&E oversight programs. 

The office of DOT&E was established because, although each of the 
Services have had their own OT&E agencies. Congress perceived that operational 
testing was not as objective as it should be. It wanted a layer of oversight over the 
procurement process, especially OT&E. As a result, DOT&E also has reporting 
requirements to Congress as well as the Secretary of Defence. [Ref. 19] 
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2. Sponsor 

The Program Sponsor is responsible for acquisition program requirements 
and related system thresholds. The requirements are determined based on continuing 
assessments of current and projected capabilities in the context of changing military 
capabilities and national Defence policy. Within the USN, the Sponsor works for the 
Chief of Naval Operations (CNO). For major warship programs the sponsor is N8 - 
Warfare Requirements. Under the new USN organisation, increasing importance is 
given to the Fleet and Type Commanders in determining requirements and 
allocating priorities. Besides generating the user requirement, the Sponsor also has 
a review function in the OT&E process. [Ref. 20] 

3. Developing Agency 

When a program achieves Milestone 0, a program office is formed to 
manage the acquisition of the system. For ships and ship systems, this office may be 
established within the Naval Sea Systems Command (NAVSEA) or is affiliated with 
a Program Executive Officer, being ‘double hatted’, with responsibilities to both the 
Assistant Secretary of the Navy (Research, Development and Acquisition), and to 
NAVSEA. The program office is responsible for the conduct of DT&E in 
preparation for OT&E. To assist all NAVSEA programs with T&E matters, 
NAVSEA has established its own T&E office (SEA 60) whose purpose is to [Ref. 
23]: 

• Assist Program Managers in complying with policies and incorporating 
lessons learned of the past. 

• Review T&E aspects of programs at major milestones 

• Keep COMNAVSEA and PEOs appraised of overall posture of T&E. 

• Influence OSD and Navy T&E policy. 
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Although primarily a DT&E policy authority, this office also has an input 
into the OT&E process by reviewing the TEMP, and by conducting OT&E readiness 
reviews of NAVSEA programs. 

4. Operational Tester & Evaluator 

The Operational Test and Evaluation Force (OPTEVFOR) is the USN’s 
sole independent operational test agency responsible for the planning and conduct 
of OT&E. COMOPTEVFOR reports directly to the CNO, and is separate and 
distinct from the developing, procuring commands and the user. Results of 
OPTEVFOR evaluations are reported directly to CNO and to DOT&E. 
OPTEVFOR’s mission is to test and evaluate weapons systems, ships aircraft, and 
equipment in the anticipated operational environment and against the anticipated 
threat: to develop and validate procedures and tactics employing these weapons 
systems, ships, aircraft and equipment: and when directed by the CNO. to assist 
developing agencies in the accomplishment of DT&E [Ref. 14:p. 1]. OPTEVFOR 
involvement in early phases of research and development includes inputs to the Test 
and Evaluation Master Plan (TEMP), observing development testing, and 
conducting those phases or operational testing necessary to provide CNO with an 
early and independent operational assessment [Ref. 13:p. 7-5]. 

5. User 

The Commanding Officer and crew of the ship under test, and his Fleet and 
Type Commanders are the ultimate users of the system. The personal assessment of 
the Commanding Officer of the ship under test is sought by OPTEVFOR during the 
conduct of OT&E [Ref. 15:p. 44]. 
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6. Coordinator 

CNO has responsibility for ensuring the adequacy of the Navy’s overall test 
and evaluation program. T&E policy and guidance are exercised through the 
Director. Test, Evaluation and Technology Requirements (N091). This organisation 
also acts as a T&E Focal Point and Coordinator, responsible for coordination of 
T&E matters in the designated Programs, System Commands and Department of the 
Navy. [Ref. 13:p. 7-5] 

E. USN OT&E POLICY 

USN OT&E policy has its basis in the statutes approved by Congress. This 
policy is refined in the DoD acquisition Directive 5000. 1 [Ref. 25] and DoD 
Instruction 5000.2 [Ref. 24]. The essential T&E elements in these directives are then 
further detailed in OPNAV Instruction 3960. 10C [Ref. 1 1J and other subordinate 
instructions. This hierarchy of documentation establishes the disciplined 
management approach to OT&E taken by the USN. 

Basically, USN test and evaluation programs are structured to [Ref. 24:p. 8-2]. 

• Provide essential information for assessment of acquisition risk and for 
decision making. 

• Verify attainment of technical performance specifications and objectives. 

• Verify that systems are operationally effective and suitable for intended use. 

• Provide essential information in support of decision making. 

OT&E programs, in particular, are structured to determine the operational 
effectiveness and suitability of a system under realistic combat conditions and to 
determine if the minimum acceptable operational performance requirements as 
specified in the Operational Requirements Document have been satisfied. U.S. 
OT&E policy requires[Ref. 24:p. 8-5]: 
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• Threat representative forces be used whenever possible. 

• Typical users to operate and maintain the system under conditions simulating 
both combat stress and peacetime conditions. 

• Production or production representative articles be used for the dedicated 
phase of OT&E that supports the full rate production decision. 

The ultimate uses of information obtained through OT&E are feedback to the 
Developing Agency on the operational strengths and weaknesses of the system, and 
inputs for decisions to proceed with production and fleet introductions. A number of 
key factors of note are highlighted in USN OT&E policy: 

1. Independence 

The fundamental DoD Directive on Defence acquisition [Ref. 25:p. 1-8] 
requires an independent operational test activity. Each Military Department is 
required to establish an independent operational test and evaluation activity. This 
activity is required to: 

• Be separate and independent from the materiel-developing and procuring 
agency and the using agency. 

• Be responsible for planning and conducting operational tests, reporting 
results, and providing evaluations of each tested system's operational 
effectiveness and suitability. 

• Report directly to the head of the DoD Component, except that the Secretary 
of a Military Department may delegate responsibility for supervising this 
activity to the Service Chief concerned. 

Acquisition managers are not to influence or attempt to influence the 
objectivity and completeness of test results presented to decisionmakers by the 
independent operational test activity. Independence is necessary so that there is no 
question of impropriety. The testing and reporting agency has nothing to gain or lose 
by whatever they say. As major acquisition decisions hinge on the results of OT&E, 
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independence is necessary to ensure that the reports on which those decisions are 
based are not self serving. [Ref. 19] 

2. Use of Contractors 

The use of system contractors in support of OT&E conducted to support a 
decision to proceed beyond low rate initial production (Milestone III) is restricted 
by law [Ref. 17] to ensure impartiality of testing. No person employed by the 
contractor for the system being tested may be involved in OT&E, unless contractors 
are planned to be involved when the system is deployed in combat. 

3. T&E Planning 

Test and evaluation planning is considered by the USN to be fundamental 
to the conduct of OT&E. The Test and Evaluation Master Plan (TEMP) is the 
controlling T&E document and is considered by some to be the single most 
important document associated with an acquisition program. The TEMP is directive 
in nature and defines and integrates test objectives, critical issues, system 
characteristics, test responsibilities resource requirements and test schedules. An 
approved TEMP constitutes direction to conduct the specified T&E.[Ref. 15:p. 6-1] 
Drafted by the Program Manager (PM), the TEMP’s main purpose is to 
combine the PM’s DT&E and COMOPTEVFOR’s OT&E into one integrated 
master plan. [Ref. 1 l:p. 11] OPTEVFOR draft Part Four on OT&E and provide 
OT&E resource requirements for Part Five. The TEMP is then co-submitted by both 
the Program Manager and COMOPTEVFOR. For ship programs, the TEMP is 
reviewed by the Assistant Chief of Naval Operations (Surface Warfare), the 
Director, T&E and Technology Requirements, and the Assistant Secretary of the 
Navy (Research, Development and Acquisition). The TEMP is ultimately approved 
at the DoD level by the Director, OT&E and the Under Secretary of Defence for 
Acquisition, Director Test and Evaluation. 
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Approval of the TEMP constitutes establishment of a contract between the 
requirement-setter and the PM which is essential to formal and accountable program 
execution, and between the requirement-setter and the OT agency, upon which 
COMOPTEVFOR will independently evaluate system operational effectiveness 
and suitability. An approved TEMP is required at Milestone I and is updated for each 
subsequent milestone and when significant program changes occur. The TEMP also 
serves several secondary purposes: 

• It allows all involved to see exactly what hurdles the system must clear. 

• It allows the PM to make good projections of COMOPTEVFOR's OT&E 
costs, which must be funded. 

• It allows fleet, range, simulator and target schedulers to plan for the required 
services. 

4. Combined DT&E and OT&E 

The USN has a policy that combined DT&E and OT&E testing should only 
be considered when there are time and cost savings [Ref. 24:p. 8-3]. However, this 
combined approach must not compromise either the developmental or operational 
test objectives. Also a final independent phase of operational testing and evaluation, 
termed “OPEVAL”, is required for beyond low rate initial production decisions 
[Ref. 24: p. 8-4]. 

“Combined testing” refers to a single test program conducted to support 
both DT&E and OT&E objectives. The advantages of combined testing are the 
shorter time required for testing, and cost savings by eliminating redundant 
activities. These need to be weighed against the limitations of the additional 
extensive coordination required and the less than optimum environment and 
coverage for OT&E that may occur. Early involvement of OT&E personnel during 
system development increases their familiarity with the system and permits 
identification of operational concerns early in the program. 
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The U.S. has extensive experience with combined testing, some successful 
and some unsuccessful, and they conclude that it is possible to have combined test 
teams involved throughout the testing process. The DT&E and OT&E teams can 
share mutually beneficial data, as long as the test program is carefully planned, 
evaluated and reporting activities are conducted separately. [Ref. 18:pp. 17-1 - 17- 
4]. 

5. Modeling and Simulation 

DoD directives encourage the use of simulation and modeling to assist in 
projecting operational effectiveness and operational suitability prior to Milestone II, 
but limit their use in subsequent OT&E to that of supplementing OT&E test data. 
The use of modeling and simulation can increase the efficiency of the T&E process, 
reduce the time and cost, provide otherwise unattainable and unmeasureable data, 
and provide more timely and valid results. Although simulations are not a substitute 
for live testing, USN OT&E policy recognises that it is useful to augment tests by 
simulating non-testable events and scenarios, and for overcoming resource 
limitations. Simulation can also be used to extend test results, to improve the 
statistical sample, or to determine overlooked or directly unmeasured parameters. 
[Ref. 18:pp. 16-1-16-8] 

6. Foreign Weapons Evaluation 

The US DoD has a foreign weapons evaluation program which is designed 
to support the evaluation of a foreign nation's weapons system, equipment or 
technology in terms of its potential to meet a specific U.S. military requirement. The 
primary objective of the program is to reduce the costs of U.S. research and 
development, while leading to the acquisition of foreign equipment for U.S. use 
[Ref. 18:p. 21-1]. From the OT&E viewpoint, the USN ‘try-before-buy policy’ is 
still maintained, despite the early phases of OT being unable to be achieved. 
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COMOPTEVFOR may be directed to assess the adequacy of any previously 
conducted OT&E and to provide recommendations on the need for additional U.S. 
T&E prior to procurement [Ref. 27:p. 6]. Obtaining pertinent test and evaluation 
data from foreign governments and manufacturers is necessary to preclude 
duplication and reduce costs. 

7. OT&E Funding 

Congress mandates that the costs of initial OT&E required to achieve a full 
production decision shall be paid from funds available for the system being tested 
[Ref. 17]. Funding associated with T&E (including instrumentation, targets and 
simulations) are identified in the system acquisition cost estimates, acquisition plans 
and the TEMP [Ref. 18:p. 18-1 1]. The Program Manager plans, programs, budgets 
and funds the costs of all resources identified in the approved TEMP for all T&E 
through OT-III. OPTEVFOR estimates the costs to conduct OT&E and the program 
manager budgets and funds these costs. OT&E costs include test articles, 
expendables, targets, data collection and reduction and OPTEVFOR program 
related costs. The Program Manager does not fund fleet operating costs for T&E 
support, which includes fuel and aircraft. The operating costs for OT-II and III, and 
all costs for OT-IV, except procurement costs and OPTEVFOR costs, are funded by 
the Fleet CINCs.[Ref. ll:p. 21] OPTEVFOR internal man-hours, and computer 
hours actually expended are not charged to individual programs. 

8. Land Based Test Sites 

A Land Based Test Site (LBTS) is a facility that duplicates, simulates or 
stimulates the employment of a system’s planned operational installation and 
utilization, primarily for the purpose of conducting DT&E. LBTS are often used to 
test system integration and overall performance. [Ref. ll:p. 12] COMOPTEVFOR 
advises the CNO on the adequacy of LBTS for the conduct of OT&E. LBTS test data 
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is normally used to support Milestone IIA decisions and not the more stringent 
Milestone III. Except where approved by CNO. OT&E intended to support 
production decisions will be performed in the operational environment in preference 
to the LBTS. 

9. Ship OT&E 

a. But Ships are Different 

The accomplishment of ship T&E varies considerably from the normal 
test cycle due to the lengthy period for design, engineering and construction of a 
major ship, and because ship T&E includes both that conducted on the ship platform 
itself, as well as that conducted on the equipment and systems to be installed on the 
ship. 

Ship acquisitions are low volume, high cost programs, and so, while 
subject to the same basic DoD and USN T&E policies applied to other systems, their 
procurement requires special T&E processes. If ships were procured the way other 
Navy systems are, the lead ship of the class would be used as a prototype for 
conducting T&E prior to approving the construction of the follow-on ships. Because 
of the time associated with the design and construction of a ship and the fact that few 
technical or operational risks are associated with the ship platform itself, the 
prototyping approach in not necessary. 

In a typical ship acquisition program, it can take about five years 
between the contract award for the lead ship and the time the ship is itself ready to 
conduct at-sea operational testing. To delay the construction of the follow-on ships 
would have a significant impact on the shipbuilding program’s costs and schedule. 
This longer schedule would substantially increase shipyard costs in return for a 
negligible reduction in the risk of the ship not meeting operational requirements. A 
significant time lag between the production of the lead and follow-on ships would 
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force a shipbuilder to let most of his experienced workers go and rehire and retrain 
others when construction resumed. Moreover, the significant cost savings available 
through quantity procurement of many of the ships’ equipments would not be 
realised if there was a time lag in production between the lead and follow-on 
ships. [Ref. 29] Also, from a contracting standpoint, a significant amount of lead 
time is required so that equipment will be available when needed for construction. 
Much of the long lead equipment is contracted before there is a ship available to 
test. [Ref. 19] 

b. Policies and Principles 

Congress accepted these practicalities of ship OT&E and amended the 
law to recognise the special case of ship OT&E [Ref. 30]. Because the development 
and construction period for a major ship normally precludes completion of initial 
OT&E on the lead ship prior to the production decision for follow-on ships, 
successive phases of IOT&E are accomplished as soon as practicable to reduce risk 
and minimise the need for modification to follow-on units. 

(1) The OT-I phase is an Eaiiy Operational Assessment (EOA) and is 
an evaluation of technologies, of processes, of plans and procedures, and design 
details to see if there any high risk areas that have a potential to cause cost overruns 
or degrade performance. 

(2) The OT-II phase for a shipbuilding program is an extension of this 
concept, except that it occurs in a time frame where landbased testing facilities or 
mock-ups of sections exist. Operability and suitability can be assessed, though not 
necessarily on a production representative item. 

Ship acquisition programs, therefore, usually have IOT&E between 
Milestones II and III, which consist of individual weapon systems testing and 
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system integration at land based test sites. A total ship OPEVAL is impractical in all 
cases except programs where a lead ship is constructed as an R&D effort involving 
a major technological advance in hull or machinery design [Ref. 28]. 



F. SHIP ACCEPTANCE 

1. Policy 

The purpose of ship acceptance in the USN is to ensure delivery to the Fleet 
of complete ships, free from both contractor and government responsible 
deficiencies [Ref. 35:p. 1]. Ship acceptance has traditionally been based only on the 
successful completion of PAT&E and material inspections, and for the first of class, 
has culminated in approval for Fleet Introduction. However, with the treatment of 
ships as complete systems, as for the DDG-51. Fleet Introduction was based also on 
OT&E results. 

2. Definitions 

Reference 35 provides the major definitions for the acceptance process: 

• “Acceptance” is defined as the legal act of accepting custody of a new 
construction ship by the Navy upon delivery of the ship by a private builder. 

• “Delivery” is defined as the actual assumption of custody by the Navy 
incident to acceptance. The date of delivery from a private shipyard is also 
the date of acceptance. 

• “Fleet Introduction” is more a concept than a specifically defined milestone l . 
It signifies approval by the Secretary of the Navy that a ship class meets the 
operational effectiveness, operational suitability, safety and material 
standards for service use. 



1. Conversation between Captain Tobin, USN, Surface Division, Board of Inspection and Survey, 
and the author, 27 October, 1992. 
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3. Responsibilities 

Independent verification of readiness of ships for acceptance is the 
responsibility of the President, Board of Inspection and Survey (PREINSURV) who 
conducts Acceptance Trials and Final Contract Trials for each individual new ship 
[Ref. 1 l:p. 10]. Based upon its findings, the Board recommends acceptance or final 
settlement of the contract. The Board identifies material conditions which represent 
departures from the General Specifications and deficiencies that substantially reduce 
the ship’s fitness for naval service and/or degrade its ability to perform its primary 
mission. [Ref. 34:p. 1] These deficiencies are noted in the Ship’s Log, but may not 
necessarily be corrected. 

A ship is normally accepted by the Commander, Naval Sea Systems 
Command. The preparation and presentation of a ship for acceptance trials is the 
responsibility of the Supervising Authority, normally the Supervisor of Shipbuilding 
(SUPSHIPS). Users also have a say in the acceptance process. A ship’s prospective 
Commanding Officer submits progress reports and can request changes be made 
which are essential to safety or the ship’s mission. The Type Commander monitors 
the construction and acceptance process to ensure “customer” input is provided, 
conducts a pre-commissioning habitability inspection and makes an acceptance 
recommendation to the Fleet Commander who, in turn, makes his recommendation 
on the request to deliver the ship. [Ref. 35:pp. 2 - 6] 

4. Procedure 

The procedure for ship acceptance and introduction to the Fleet is outlined 
in Figure 5. The Total Ship Test Program outlines two phases of this process [Ref. 
32: p. 2-6]: 

• Ship Construction Tests and Trials consisting of all testing conducted during 
construction, including Acceptance Trials. 
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• Ship Post-Delivery Test and Trials include final contract trials that 
commence after ship delivery and continue to the end of the ship construction 
obligation. 




Figure 5: Traditional Ship Acceptance Schedule and Milestones 



This procedure includes the following activities and milestones: 
a . Delivery / Acceptance 

Acceptance Trials are conducted by the INSURV Board when all work 
has been completed. These trials are conducted at sea and in port when the following 
prerequisites are met [Ref. 35:p. 11]: 

• Successful completion of builders trials. 

• All installed equipment operable and capable of meeting performance 
specifications. 

• Habitability items complete. 

• Completion of surveys not requiring remote ranges. 

• Installation and checkout tests completed. 
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• All applicable naval certifications completed. 

• Completed test reports and certifications available for inspection by the Trial 
Board. 

On successful trial completion, and on the recommendation of 
PREINSURV, COMNAVSEA accepts the ship on behalf of the USN from the 
contractor. 

b. Final Contract Trials 

Final Contract Trials are trials and materiel inspections conducted at 
sea and in port by the INSURV Board to determine if builder responsible 
equipments are operating satisfactorily [Ref. 35:p. E3]. It involves operation of all 
combat systems, and the propulsion system at full power. On completion, 
PREINSURV submits a technical assessment of readiness for OT&E to CNO, and 
to COMOPTEVFOR when tasked by CNO [Ref. 1 1 :p. 10]. 

c. Fleet Introduction 

The successful completion of Final Contract Trials on the first ship of 
the class traditionally results in a recommendation for Fleet Introduction by 
PREINSURV to CNO. The Secretary of the Navy then approves Fleet Introduction 
of the ship class on the recommendation of CNO. In contrast. Fleet Introduction of 
systems is traditionally recommended by OPTEVFOR after successful completion 
of OT&E. However, for the DDG-51 class, where the ship was viewed as a complete 
system. Fleet Introduction was recommended by both PREINSURV and 

OPTEVFOR 2 after the completion of OT&E. Figure 6 show the acceptance and 
schedule milestones used for DDG-51. 



2. Conversation with Commander J. Kren, USN, OPTEVFOR, and the author 29 October, 1992. 
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Figure 6: DDG-51 Ship Acceptance Schedule and Milestones 



5. Ship Acceptance and OT&E 

OT&E has traditionally not had an input into the ship acceptance process in 
the USN, with no involvement in Delivery, Acceptance or Fleet Introduction. It 
primarily focussed on the ship’s systems rather than the ship as a whole, and was 
fundamental to the Fleet Introduction of these systems. With the view of the DDG- 
51 as a complete system, successful completion of OT&E is now a prerequisite for 
the Fleet Introduction of all new warship classes. 

G. USN OT&E PROCEDURES 

Procedures are the methodology by which policy is carried out. This section 
discusses the procedures adopted by the USN in implementing OT&E policy. 

1. From the User Requirement to the OT&E Test Plan 

The determination of the level and extent of OT&E required is a difficult 
question. For their multiwarfare capable ships, the USN focuses on the mission 
warfare areas to determine what is appropriate [Ref. 19]. The fundamental 
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cornerstone of OT&E is the user requirement, generated through a series of 
documents built into the acquisition process. The procedure involves progression 
through the following documents (Figure 7): 

• Mission Need Statement (MNS) 

• Cost And Operational Effectiveness Analysis (COEA) 

• Operational Requirements Document (ORD) 

• Top Level Requirement (TLR) 

• Test and Evaluation Master Plan (TEMP) 

• OT&E Test Plan (OT&E TP) 
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Figure 7: Progressive Definition of OT&E Documentation 
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a. Mission Need Statement 

Acquisition programs are based on identified mission needs that are 
generated as a direct result of continuing assessments of current and projected 
capabilities in the context of changing military threats and Defence policy. A broad 
statement of mission need, expressed in terms of an operational capability, not a 
system-specific solution, is identified in a Mission Need Statement (MNS).[Ref. 
24: p. 3-2] 

b. Cost and Operational Effectiveness Analysis 

A cost and operational effectiveness analysis (COEA) may be 
undertaken to evaluate the costs and benefits of alternative courses of action to 
satisfy the requirements of the MNS. The COEA looks at other alternatives e.g., are 
a lot of little ships or a few big ships needed to do the mission? The most suitable 
alternative is selected, e.g., a lot of little ships are required, and they need to do these 
things. It also develops the Measures of Effectiveness (MOE) for assessing the 
operational effectiveness of the alternatives. 

c. Operational Requirements Document 

From the MNS and COEA is developed the Operational Requirements 
Document (ORD) that contains the performance (operational effectiveness and 
operational suitability) and related operational parameters for the proposed concept 
or system. It contains the system performance objectives and minimum acceptable 
requirements to meet the requirements of the MNS. The MOEs used in the ORD and 
the technical parameters, are designed to support those same MOEs that were 
developed in the COEA [Ref. 19]. 
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cL Top Level Requirement 

For some ship acquisition programs. Top Level Requirements are 
prepared after the MNS and ORD. This is necessary because of the length and 
complexity of the ship design process. The TLR states the high level requirements 
for the ship. For example, the TLR for the DDG-51, Section 2.11a Operational 
Directions, under Mission, Warfare Areas states: 

Simultaneous action in Strike Warfare, Antisubmarine Warfare, Antisurface 
Warfare and Antiair Warfare required. The combat system shall provide for 
rapid, accurate and efficient employment of the ship’s weapons. 

e. T est and E valuation Master Plan 

The TEMP is developed from the ORD, and TLR if produced, and is 
the controlling T&E document that defines and integrates the test objectives, and 
Critical Operational Issues (COI) based on the ORD. For example, the DDG-51 
TEMP [Ref. 31:p. IV-2] highlights the COI of Combat System Performance as 
follows: 

Will the combat systems support simultaneous action against threats in Antiair 
Warfare (AAW), Antisubmarine Warfare (ASW), Antisurface Warfare 
(ASUW), and Strike Warfare (STW) missions during independent or 
combined operations as a unit of a Battlegroup, Surface Action Group, 
Underway Replenishment Group or Amphibious Task Force? 

This COI is addressed as a specific objective for OT-IIIA [Ref. 31:p. 
IV - 1 1] which is to: 

Determine the capability of the integrated DDG-51 combat system to detect, 
track, localise and engage threat representative targets in a multiwarfare 
environment (STW, ASW, ASU and AAW). 
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From this specific objective, a test procedure is outlined by 
OPTEVFOR. The level and extent of testing is based on the level of confidence to 
be assessed. It is usually a compromise between the extent of testing and funding 
available, agreed between the PM and OPTEVFOR with DOT&E oversight. The 
TEMP is viewed as a contract between the Developer, the Sponsor, OPTEVFOR 
and OSD, through DOT&E, as what is committed to be done and funding is 
available. Any change to the TEMP must be approved by all signatories. The TEMP 
identifies limitations to the scope of testing which may include threat replicators 

e.g., when a MIG-29 threat aircraft is needed during testing, but an F-14 will be 
used. [Ref. 19] 

f. OT&E Test Plan 

This specific objective is then followed through to the OT&E Test Plan, 
developed by COMOPTEVFOR. The test plan contains “E” tests that assess 
operational effectiveness and “S” tests that assess operational suitability.The test 
plan then addresses how this will be done, by breaking it down into more specifics. 
AAW, EW, CAP, CIWS. Under DDG Test E-l Combat System Performance, the 
Test Plan states: 

To determine the capability of the integrated DDG-51 combat system to 
detect, track, localize and engage threat representative targets in a 
multiwarfare environment (STW, ASW, ASU and AAW). 

g. Summary 

A requirement can, therefore, be traced from the mission need to the 
ORD, through to a COI and specific a objective in the TEMP, to the test plan and 
ultimately to the final OT&E report by OPTEVFOR. This linkage from mission 
need to operational test program defines the scope of testing. If a ship has a number 
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of mission areas and there are MOEs to evaluate the performance in those mission 
areas, then a test program is developed around those MOEs. 

2. Assessment of Operational Effectiveness 

In assessing operational effectiveness, OPTEVFOR examines each COI 
and related operational effectiveness objective, and decides what needs to be known 
to enable each objective to be assessed [Ref. 15:p. 9-1]. For example, the objective 
may be “to determine the sonar’s capability to detect, classify, and track... in the 
natural acoustic environment.” OPTEVFOR may decide the following measures of 
effectiveness are needed to assess the overall objective: 

• Probability of detection 

• Detection range 

• Probability of correct classification given detection 

• Probability of classifying a threat as a nonthreat 

• Time between detection and classification 

• Classification Range 

• Probability of establishing a track, given detection 

• Time between detection and track establishment 

• Range at track establishment 

• Percent of time tracks are held 

They ensure that the appropriate environments, threats, etc., are included 
and that sufficient data will be generated to address the COI and objectives. 
OPTEVFOR initially determines the requirements and resources necessary to 
conduct the test. This is achieved by making a list of environments and other 
conditions under which the system will operate, and associating them with the 
various threats. This can be achieved by constructing a test design matrix of which 
Table 5 is an example. 
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TABLE 5: SAMPLE TEST DESIGN MATRIX [Ref. 15:p. H-2] 
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The letters indicate missile firing runs in alphabetical order of priority. The 
columns are the environment in which the firing is scheduled. [Ref. 15:p. H-l] Then 
the actual resources available are reviewed to determine which test objectives must 
be eliminated due to lack of funding. 

Even though the operational test program is linked to mission need through 
the COEA, ORD and TEMR there are a number of difficulties in developing 
appropriate OT&E tests. Many operational requirements are not very specific, 
meaning the testing and subsequent assessment of those requirements may be very 
subjective, and critical definitions need to be agreed before testing commences. 
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a. Qualitative Vs. Quantitative Assessment 

The broad scope of many operational requirements makes their testing 
and subsequent assessment rather subjective, making the determination of 
meaningful and assessable quantitative MOEs difficult. Also a lack of resources 
may reduce an objective, quantitative, and statistically relevant test schedule to a 
more subjective, qualitative assessment. These situations require expertise and 
judgement, and compromise between the authorities involved, to make the tests as 
objective as possible. As an example, an antisurface warfare mission may require 
the ability to sink another ship, but the test program does not include this provision. 

The test program has to be designed so that it will provide confidence 
that the ability can be demonstrated without actually sinking a ship. Similar 
situations are encountered in all the warfare areas, and additionally with ship 
survivability. One objective is to achieve statistical relevance. DOT&E works 
closely with the USN to help determine how many test assets are required to achieve 
a certain degree of confidence that a test objective has been met. A test may need to 
be repeated a number of times to gain statistical relevance. However, if this is not 
possible, or the direct proving of an ability is not possible (as in the surface warfare 
example), then the test needs to be designed so that what is observed in the test 
program is projected to be what would be observed in reality. 

OPTEVFOR indicates the statistical significance of a result as follows: 

• “Determine” means a statistically significantly result with a specified level of 
confidence. 

• “Assess” means to qualitatively evaluate which will not achieve a 
statistically significantly result. 
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b. Definition Agreement 

Another difficulty in assessing operational effectiveness is in defining 
the meaning of the requirement and the measures of effectiveness. This is 
particularly applicable to whole ship testing which may incorporate a specific 
equipment requirement in addition to an overall ship requirement. For example, the 
definition of a “Detection Opportunity” for a sonar, may differ from what is an ASW 
“Detection Opportunity” for the ship, given the other ASW related sensors that may 
be available. Agreement between authorities and the MNS is required for the whole 
ship versus individual systems. [Ref. 21] 

3. Assessment of Operational Suitability 

The assessment of operational suitability in the USN is more standardised 
than for the assessment of operational effectiveness. OPTEVFOR identifies 14 
suitability issues for which they have standard suitability tests [Ref. 15:p. 9-4 - 9-8]: 

• Reliability 

• Maintainability 

• Availability 

• Logistic supportability 

• Compatibility 

• Interoperability 

• Training 

• Human factors 

• Safety 

• Documentation 

• Transportability 

• Wartime usage rates 

• Manning 

• Software supportability 
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Software supportability has only recently been added by OPTEVFOR. It 
relates particularly to totally intensive computer systems, termed “software 
intensive systems”. It determines if a program manager has established adequate 
software support for his system. [Ref. 21] Not all these suitability issues need be 
addressed for a particular system. The nature and use of the system, and the phase 
of OT&E will determine the issues to be assessed. 

a. Suitability Assessment Procedure 

Early in the acquisition program, OPTEVFOR reviews the Integrated 
Logistic Support Plan (ILSP) against the ORD. The ILSP is a key acquisition 
document which defines the various methods used to provide the required range of 
logistic support for the system.Following ILSP review, OPTEVFOR determines the 
suitability tests required based on expected reliability, degree of confidence, 
thresholds, etc. required to be achieved. These determine the scope and length of the 
assessment. An onboard assessment at sea is conducted, during which 
questionnaires of crew members may be used and maintainability demonstrations 
are performed, usually using pre-faulted modules. Reliability, maintainability and 
availability (RM&A) data is collected and sent to the normal inservice analyst of 
RM&A data, the Naval Weapons Analysis Centre (NWAC) for analysis. In later 
OT&E, OPTEVFOR also uses data from the inservice maintenance management 
system (3M) and the individual system logs. [Ref. 21] 

Not only does OPTEVFOR have specialist analysts for effectiveness 
issues, it also has suitability analysts who design tests and determine measures of 
suitability. They also evaluate and analyse the adequacy of logistic supportability. 
OPTEVFOR also relies on operational personnel to use their experience and 
knowledge of the system to identify inadequate logistic support. 
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4. OT&E Reporting 



a. Reporting by COMOPTEVF OR 

COMOPTEVFOR’s evaluation report provides the CNO with 
conclusions regarding a system's operational effectiveness and operational 
suitability, and his recommendations regarding the systems future, i.e., fleet 
introduction, further development, additional OT&E, etc. The report also contains 
the test results and evaluation criteria to substantiate the conclusions and 
recommendations. [Ref. 15:p. 12-1] A report covers a complete OT&E phase (e.g., 
OT-IIA), relating the test results to the COIs and addressing the objectives stated in 
the TEMP. COMOPTEVFOR requests comments from the Commanding Officer of 
the ship under test ship. These comments are sent only to OPTEVFOR. All 
operational test data is considered to be the owned by OPTEVFOR until the final 
report is signed. Quick-look and interim reports are usually sent only if the testing 
could not be completed or when directed by the CNO. Although the final report may 
be a surprise to other authorities, the USN appears to favor this reporting procedure 
to ensure that: 

• the conduct of OT&E remains impartial and is not influenced by the program 
manager or contractors during testing and analysis. 

• OPTEVFOR’s conclusions are based on a complete and thorough analysis. 

• OT&E results and analysis are reported through one authority only. 

b. Reporting by DOT &E 

The reports rendered by DOT&E address whether the OT&E 
performed was adequate and whether the OT&E results confirm that the system 
actually tested is operationally effective and operationally suitable. In the past it 
used OPTEVFOR reports as basis and provided their own judgement. Following 
General Accounting Office (GAO) criticism that DOT&E overall assessments 
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consistently presented a more favorable presentation to the Congress of test 
adequacy and system performance than was warranted by the facts [Ref. 36:p. 31], 
it now does an independent analysis from the raw test data collected by 
OPTEVFOR. As DOT&E does not have any specialist analysis staff, it hires an 
independent contractor who works exclusively for DOT&E, to do the analysis. 
Experienced operators at DOT&E then make the assessment. DOT&E's written 
report to Congress is based now on OPTEVFOR reports and its own independent 
analysis. [Ref. 21] 

5. OT&E Coordination 

The USN recognises the importance of coordination between authorities in 
the successful achievement of T&E. The USN T&E coordinator is responsible to the 
Director, Test, Evaluation and Technology Requirements (N091). He is essentially 
the OPNAV area coordinator for Navy T&E, providing a primary contact point for 
all parties and setting up T&E briefings and meetings. His responsibilities include 
the chairmanship of the Test and Evaluation Coordination Group (TECG) for each 
major program. [Ref. 37] TECGs are used for complex, multifaceted programs 
which require extensive T&E coordination. Membership of a TECG includes the 
Program Manager, the Sponsor, COMOPTEVFOR, a logistics coordinator and 
others as appropriate (such as a PREINSURV representative). TECG 
recommendations are considered for inclusion in the TEMP.[Ref. 1 l:p. 15] Some of 
the functions of a TECG are the early definition of terms, measures of effectiveness 
and how these are to be measured, and the criteria for acceptable or not acceptable. 
Of note is the formation of a TECG does not imply a joint test team approach. Each 
T&E agency remains fully and solely responsible for conducting and reporting the 
types and phases of T&E for which it is accountable. 
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6. OT&E Personnel Selection and Training 

OT&E is a specialised discipline, with its own philosophy and 
methodology. The U.S. community involved with ship OT&E select and train their 
military personnel. Although some personnel may develop into OT&E specialists, 
no career path in OT&E is consciously provided, and no naval personnel currently 
serving in DOT&E have served previously in OPTEVFOR. 

a. OPTEVFOR 

There are no special selection requirements for detailing military 
personnel for service as Test Directors in OPTEVFOR, although broad ship 
operations experience and combat system knowledge are preferred. OPTEVFOR is 
functionally organised along warfare lines, so specialists in each warfare area are 
preferred. Most Test Directors are of Commander rank, reporting to Captain level 
section heads. For particular programs, OPTEVFOR makes use of subject area 
experts, either resident or borrowed from a non-interested party. OPTEVFOR 
conducts an intensive four day Operational Test Directors (OTD) course covering 
the major areas of OT&E. Further details of this course are included in Appendix A. 
They also run adhoc segment courses which provide acquaints or updates on OT&E 
subjects e.g., analysis, test plan development and threat updates. To assist their 
personnel in managing OT&E, OPTEVFOR publishes the OT&E Director’s Guide 
which documents its philosophy and methodology. [Ref. 21] 

b. DOT&E 

Military personnel assigned to DOT&E are selected by background and 
expertise, and are preferably war college graduates with joint experience. They are 
interviewed by their potential immediate superior, and by DOT&E himself, before 
their appointment is confirmed. DOT&E is platform, rather than functionally 
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organised, so specialist surface warfare, aviators and submariners are preferred. 
Although DOT&E does not conduct courses for new personnel, a one month 
handover is usually required. It does, however, publish a comprehensive staff 
orientation guide and some personnel have attended the OPTEVFOR OTD course. 
[Ref. 19] 

H. USN OT&E IMPLEMENTATION - THE DDG-51 PROGRAM 

The objective of the DDG-51 Program is to build the next class of destroyer to 
replace the aging ‘Coontz' (DDG-37) and ‘Charles F. Adams' (DDG-2) classes. The 
first of the class, the USS ARLEIGH BURKE has recently completed OT-III. It is 
the latest USN surface warship to undergo OT&E, thus serving as the most recent 
example of the implementation of USN OT&E policy and procedures. The DDG-51 
Program is under the management of the AEGIS Shipbuilding Program Manager 
(PMS 400) and is sponsored by the AEGIS Program Sponsor (N865G). PMS 400 is 
chartered to provide comprehensive direction and program management for all 
aspects of system development, ship acquisition and lifetime support preparation. In 
this task, PMS 400 has the traditional USN responsibilities of a ship acquisition 
program manager and also bears broadened responsibilities for the fleet introduction 
and lifetime support of the DDG-51 class ships. [Ref. 33 :p. 1] 

1. Critical Issues 

The Critical Operational Issues for the DDG-51, as defined in the TEMP, 
are subdivided into effectiveness and suitability issues. [Ref. 31:p. IV-2] The 
effectiveness issues for DDG-51 are: 

• Combat System Performance 

• Mobility System Performance 

• Command, Control and Communications Performance 



59 



• Support System Performance 

• Survivability 

• Tactics 

The suitability issues considered critical for DDG-51are: 

• Reliability 

• Maintainability 

• Availability 

• Logistic Supportability 

• Compatibility 

• Interoperability 

• Training 

• Human Factors 

• Safety 

• Documentation 

• Software Supportability 

2. DDG-51 OT&E Program 

These COIs were assessed in a multi-phased OT&E program as shown in 
the schedule in Figure 8. 



a. Initial OT&E 

OT-II was split into two major system areas. Propulsion System and 
Combat System and was conducted at Land Based Test Sites. 

(1) OT-IIA Propulsion System Testing was conducted on the DDG-51 
Propulsion System at the Gas Turbine Ship Land-Based Engineering Site 
(GTSLBES), Naval Ship Systems Engineering Station (NAVSSES), Philadelphia, 
Pennsylvania. It is an Engineering Development Model that includes the Machinery 
Control System, Data Multiplex System and the engines, reduction gears, thrust 
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bearings and propulsion-related auxiliaries of the complete DDG-51 system. A 
water brake was used to simulate the propeller. The GTSLBS was exercised in a 
simulation of the operational environment of DDG-51class ships. Casualty control 
drills, maintenance demonstrations, and individual ship propulsion events were 
conducted during 424 hours of test operations.These included full power runs and 
emergency crash astern maneuvers. The system was assessed to be potentially 
operationally effective and potentially operationally suitable. [Ref. 31:p. IV -5] 




Figure 8: DDG-51 Integrated Test Program Schedule [Ref. 31] 



(1) OT-IIB Combat System Testing was conducted at the Combat 
System Engineering Development Site (CSEDS) Moorestown, New Jersey. 
Following a philosophy of build a little, test a little, OT-IIB was conducted in three 
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phases. OT-IIB-1 was conducted in Jun 1986 on a partial configuration in an early 
stage of development and integration to support the program Milestone III A 
decision. The capability in all mission areas could not be evaluated with only search, 
detection and tracking of AAW targets conducted. No engagements of air targets 
were conducted. Twelve raids of multiple aircraft were conducted, of which eleven 
were in an ECM environment, including various jamming modes and chaff. 
Dynamic Test Targets (DTT) were also introduced to evaluate system performance. 
COMOPTEVFOR concluded that the combat system demonstrated the capability to 
control, integrate, and display information required for tactical decisions on AAW 
targets and recommended approval of the DDG-51 Combat System for limited fleet 
introduction. [Ref. 31:p. IV-4] 

(2) The second phase (OT-IIB-2) was conducted in September 1988 
when the CSEDS represented an operational prototype replicating planned human 
factors, sensors, information processing and weapons system installations in the real 
DDG-51. Assessment of capability in all mission areas was conducted. Thirty-four 
raids of manned aircraft were conducted, of which two only were in a clear 
environment. Additionally DTTs were also introduced to represent antiship missiles 
and surface hostile threats. Threat submarine evaluations, both active and passive 
were conducted as were Link 1 1 operations with aircraft, surface ships and shore 
stations. Non firing engagements with both Harpoon and Tomahawk were also 
conducted. Operational Suitability data were collected during the period. On test 
completion, COMOPTEVFOR recommended continued development of the 
Combat System and the conduct of an additional phase of OT-II at CSEDS prior to 
sea trials. 

(3) The third phase (OT-IIB-3) was conducted in August 1990 
following further development of the Combat System. The testing was 
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comprehensive of the total Combat System’s capabilities. COMOPTEVFOR noted 
that performance was notably improved since the conduct of OT-IIB. They assessed 
the system as being potentially operationally effective and potentially operationally 
suitable, and recommended continued development of the Combat System. [Ref. 
31:p. IV-6] 

OT-IIB-2 and OT-IIB-3 were conducted to support continued limited 
production. DDG-51 crew members participated in the OT-II testing, both at the 
GTLBES at NAVSSES and in CSEDS testing. 

b. Follow On OT &E 

(1) OT-IIIA was conducted for a 28 day period in early 1992 on board 
USS ARLEIGH BURKE. Air, surface and subsurface targets were presented in 
single and multithreat environments. Simulated firings against manned aircraft and 
live firings against threat-representative targets in operationally realistic scenarios 
were conducted in order to evaluate the performance of the integrated combat 
system and the capability of the mobility and support systems to support the ship’s 
mission. The performance of all the ship systems designed for operation with other 
Navy units was also evaluated. Suitability tests, off-ship support and maintainability 
demonstrations were also conducted. [Ref. 31:p. IV-9] 

(2) OT-IIIB is planned to be conducted in a DDG-51 class ship to 
demonstrate correction of deficiencies identified during OT-IIIA and to include 
previously untested systems. 

(3) OT-IV will be conducted in a DDG-51 class ship to demonstrate 
correction of deficiencies, to complete deferred or incomplete OT&E, and to 
evaluate major computer software revisions as well as the DDG-51 class 
performance in battle group operations. 
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c. Timing of OT -III 



Table 6 shows the Post Delivery Schedule of DDG-51. Of interest is the 
timing of OT-III within the ship’s program. Note that OT-III is scheduled 
immediately after DT-III testing and before the Workup (REFTRA and FLEETEX). 
This has the disadvantage of OT&E not being conducted on a fully worked up ship 
with the crew still Developmental Testing oriented. This may have prejudiced the 
OT&E results. 



TABLE 6: USS ARLEIGH BURKE - POST DELIVERY SCHEDULE [Ref. 33:p. 5] 



Time 


Activity 


D + 1 month 


Mobile Training Team / Training Readiness Evaluation 


D + 2m 


Light Off Examination / Fast Cruise / Commissioning / Port Visit 


D + 3-4 m 


Combat Systems Ship’ s Qualification Trials (CSSQT) 


D + 5 m 


DT-III (Demonstrate ship performs as designed) 


D + 6 m 


Final Contract Trials / OT-III 


D + 7 m 


OT-III / Operational Propulsion Plant Examination 


D+ 8-10 m 


Post Shakedown Availability (PSA) 


D+ 11 m 


Refresher Training (REFTRA) 


D+ 12 m 


Fleet Exercise (FLEETEX) 


D+ 13 m 


Upkeep (Maintenance Period) 


D + 14 m 


OT-IV (Backup Events) 


D + 15 m 


FLEETEX / Battle Ready 


D + 16 m 


Pre Overseas Movement / Deploy 
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3. ‘Whole Ship’ OT&E 

OT-III conducted on the DDG-51 was unique in that it was the first time the 
USN has conducted OT&E on a whole ship in a multi-battle situation, in a fairly free 
play environment. Previous ship OT&E, e.g., on the “Aegis” class cruisers (CG-47), 
was essentially a combat system test based on a proven hull, where the rest of the 
ship and ship integration were not tested. DOT&E have been instrumental in 
intensifying efforts to create an operational environment. 

a. OT&E Program Development 

Following the guidance contained in the OTD manual [Ref. 15], 
OPTEVFOR developed a test design matrix. This matrix mapped the Top Level 
Requirements to the Schedule of Events (SOE). The TLRs were general and related 
to the warfare areas of AAW, ASW, ASUW and STW. The purpose of the matrix 
was to ensure that all TLRs were covered by an operational test. In the event that test 
resources became unavailable or a system casualty occurred, the matrix provided a 
reference of what had been accomplished, what remained to be accomplished, and 
the feasibility of completing the test objectives if OT was to continue. In the 
evaluation period, the “as-run SOE” was laid out chronologically, identifying events 
by mission areas. By examining the as-run table, individual mission areas were 
identified for analysis and periods of multi warfare could be identified to look at 
synergistic effects. [Ref. 21] 

b. Integration of Engineering and Combat System Events 

Whole ship testing includes engineering and technical problems 
combined with combat system activity. The OPTEVFOR Operational Test Director 
controls the program and controls injection of incidents. A balance must be achieved 
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between engineering incidents and related degradation of the combat system to 
ensure the overall aim of the test is achieved. 

c. Operational Realism 

The OT&E emphasised operational realism by testing the ship as 
though in war, with a full political background to set tone and flavor. The situation 
consisted of an overall 28 day mission consisting of individual, but interlinked, 
scenarios. The scenarios were developed to examine the ship’s capability to operate 
as a fleet asset and to stress the installed combat systems’ capabilities against the 
threat. They were designed to provide a realistic threat in an operational 
environment and an element of surprise to provide realistic tests to support 
completion of the OT&E objectives and resolutions of the COIs. The Commanding 
Officer, Combat Information Centre team and the ship’s crew were allowed to 
respond to the tactical situation as they perceived it.[Ref. 31:p. IV-9] To enforce 
realism, operational message traffic flow at the same rate that could be expected in 
a conflict was maintained and Damage Control incidents were introduced. 

d. Limitations to Scope of Testing 

Despite the efforts for realism, certain practical considerations, time 
restrictions and resource constraints limited the scope of operational testing. These 
were [Ref. 31:p. IV- 10]: 

• Targets and ECM did not fully replicate the threat in numbers and 
characteristics thus precluding fully threat representative raids. 

• The CIWS, ESM and ECM were not tested using drone targets. Manned 
aircraft were used for ESM and ECM tests and a towed target was used for 
CIWS firings. 

• The number of missiles available precluded reengagements and multiple 
salvo engagements in some scenarios. 

• The geographical location of test ranges and their facility and equipment 



66 



limitations precluded fully threat representative target presentations, target 
density and open ocean geometries. 

• The duration of the operational testing precluded determination of reliability 
and other suitable parameters, forcing the use of development testing data 
where credible and applicable. 

• Operations were not conducted in all weather, temperature, humidity, and 
wind and sea conditions. 

e. DDG-51 OT&E Analysis 

The OT&E of DDG-51 highlighted a number of overall 
considerations. [Ref. 21] 

(1) Interfaces. Interfaces are important to consider in whole ship 
testing, particularly individual systems with their own TEMPS and test plans. Whole 
ship OT&E looks at whole mission system and ship performance e.g., not just sonar 
performance but ASW system performance. Problems occur with concurrent sub- 
system and whole ship testing due to different critical issues and objectives of 
testing which create conflicts during testing. One overall test plan resolving these 
issues is required. 

(2) Statistical Significance. Some tests, e.g., Tomahawk firing, can 
only be done once. To achieve results of statistical significance, a number of 
simulation runs were conducted using new and varied scenarios. The one firing was 
then used to verify the simulation. This demonstrates the use of simulation to extend 
the test results. 

(3) Qualitative vs. Quantitative. Effort was made throughout DDG-51 
OT&E to use quantitative thresholds based on quantitative numbers, whereas, 
previous OT&E of the AEGIS class cruisers (CG-47) qualitatively assessed all 
COIs. 
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(4) Operationally Realistic Scenarios. To improve the measurement of 
AAW mission performance against a threat representative target in a threat 
operational scenario, particularly in the light of the “Iranian Airbus” issue, AAW 
scenarios now do not necessarily include everything that flies is hostile. To add to 
the operational flavor, a political situation and Rules of Engagement (ROE) are used 
during OT&E. These ROEs change due to the political situation or opposing force 
actions e.g., switching on fire control radar. This level of testing adds to the analysis 
problem e.g., having to cross the Electronic Warfare detection of a Fire Control 
radar to the aircraft, with the subsequent need for overall exercise analysis. 

(5) Crew Stress under Battle Conditions. The DDG-51 OT&E 
program include the measurement of crew stress under battle conditions, both 
during the land based and at sea testing. They observed that the land based level of 
operator performance was significantly better than at sea. The USN is continuing to 
pursue methods to take measurements to determine how much of the performance 
degradation from land based testing to at sea testing is a result of the personnel. The 
man machine interface, the concept of human factors and operational stress are a 
great contributor to overall system performance. One objective of the OT&E 
program was to keep the crew busy during the test period. 

(6) System Problems vs. People Problems. One aim of OT&E is to 
assess the human factors relating to system performance which involves assessing 
the adequacy of training, ease of use, etc. It does not include the performance 
assessment of an individual. This can sometimes be difficult when people form a 
vital part of the system. OPTEVFOR stress they are not drilling the crew, but 
demonstrate how the system (which may include people in the loop) responds. It is, 
therefore, important for the crew to be ‘worked up” to normal fleet standards prior 
to OT&E. 
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(7) Independent Third Party Data. OT&E needs to be conducted on a 
range or with another ship/aircraft to get independent third party data to be able to 
reconstruct the evolution. The philosophy of testing is that the item under test is 
immediately suspect so if data from this system is required, then it needs to be 
verified. It may be sample verified so that 100% duplication is not required. 

(8) Program Manager Perception. Although many program managers 
may see OT&E as an annoyance and unnecessary burden to their program, PMS 400 
sees OT&E as an independent assessment to demonstrate that its on the right track 
and doing the right thing. [Ref. 22] 

I. U.S. DEPARTMENT OF DEFENCE OT&E ISSUES 

U.S. DoD authorities involved in ship OT&E see a number of macroscopic 
issues relating to OT&E in the USN today. 

1. Impact of Budget Cutbacks 
a. Acquisition Strategy 

As a result of budget cutbacks, the Secretary of Defence is developing 
a new acquisition policy for systems that don’t go into a long protracted production 
phase. A prototyping acquisition strategy may be implemented where one of kind is 
built and the technology is either put on the shelf, or into limited production to keep 
defence industry active. More emphasis may be placed on R&D than on production. 
Questions arise as to how OT&E fits into an acquisition strategy that does not lead 
to production, and should more reliance now be placed on DT&E and less on 
OT&E. [Ref. 19] Current thinking is that more importance may be placed on 
modeling and simulation to estimate operational effectiveness and operational 
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suitability of advanced technical demonstrators, to minimise the operational risk if 
production and deployment is required in an emergency. 

b. OT &E Conduct 

Questions arise also of how OT&E is to be conducted in a reduced 
budget environment. With reduced production runs, the relative cost of OT&E in 
relation to overall program cost will increase. This may not be seen as acceptable 
and pressure may be applied to reduce the absolute cost of OT&E. The breadth of 
assessment may be diminished using less test assets. There may be also a possibility 
of a central OT&E agency. With DOT&E becoming an increasingly aggressive 
participant in OT&E, testing in the future could be directed exclusively by 
DOT&E. [Ref. 19] 

c. Increasing Modeling and S im ulation 

Modeling and simulation (M&S) capabilities are going to play a larger 
role in contributing to the OT process. Although M&S will never replace operational 
testing, it will be able to supplement actual testing to a greater degree in certain 
areas. Technological advances in M&S may make these tools now more of a 
candidate to be used for OT&E. Typically M&S is more of a study or analysis or 
developmental type tool, but are rapidly developing capabilities that will offer 
advantages to the OT process. [Ref. 19] [Ref. 21] 

2. Operational Realism 

Following GAO criticism of lack of realism [Ref. 36] and the ‘whole ship' 
OT&E concept, test scenarios can be expected to be more realistic in the future. To 
obtain the maximum benefit from operational tests, they will also need to be more 
quantitative. This may include the development of integrated multiwarfare ranges to 
support whole ship testing. [Ref. 19] [Ref. 21] 
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J. USN OT&E SUMMARY 



1. Importance of OT&E 

OT&E is the “final exam” in the U.S. DoD acquisition process, an 
empirical method of ensuring that sufficient technical return is made on acquisition 
investments and that a new system is fully capable of meeting the Fleet’s needs prior 
to production [Ref. 12]. As a result, the U.S. place high importance on OT&E. The 
requirement for the conduct of OT&E is mandated by Congress and incorporated in 
the laws of the U.S. OT&E is viewed as being more important as a basis for a 
decision to proceed beyond LRIP, than on the introduction into service of the final 
production item. This is because the “big bucks” of most Defence acquisitions are 
spent during the production phase. OT&E, however, continues to play an important 
role in the assessment of systems for Fleet Introduction and has increased its 
importance in the assessment of complete ships for Fleet Introduction. 

2. Need for OT&E 

The USN need for OT&E stems from the almost exclusive reliance by the 
U.S. on indigenous weapons development and production. They have a history of 
large, risky, developmental programs which push the state of the art in weapons 
technology. The outputs of these programs require the assessment of operational 
effectiveness and suitability before being committed to production and subsequent 
introduction into the fleet. The U.S. has been involved in a number of military 
conflicts which demonstrate the need for operationally effective and suitable 
systems. The lessons learned from Vietnam, and Gulf operations including the USS 
STARK and USS VINCENNES incidents, highlighted problems which DT&E 
alone may not identify. 
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3. OT&E Agency Independence 

Independence is the key to the effectiveness of OT&E in the U.S. 
acquisition process. OPTEVFOR is the sole independent OT&E agency within the 
USN, reporting directly to CNO, and DOT&E provides independent oversight and 
coordination of the military services' planning and execution of OT&E, reporting 
directly to the Secretary of Defence. OT&E agency independence is designed to 
ensure impartiality, honest and open reporting, with a minimum of political 
interference. 

4. T&E Delineation 

The USN have a clear delineation between the different types of T&E. 
OPTEVFOR conduct only OT&E, and OT&E within the USN is conducted solely 
by OPTEVFOR. Thus, there appears to be good understanding in the USN of what 
is, and what isn’t, OT&E. 

5. Whole Ship OT&E 

Whole ship OT&E is conducted in a dedicated period following CSSQT 
and Final Contract Trials, and consists of fully assessed free play, multi-threat 
scenarios. The whole ship is viewed as an integrated warfare system, rather than as 
a platform for warfare systems. 

6. Clear Guidance 

The requirement for OT&E, and its organisation and methodology are well 
documented within the USN, providing clear guidance to acquisition and operations 
personnel. Also, training in OT&E is provided, further improving the capability and 
understanding of personnel involved with OT&E. 
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7. Summary 

The USN OT&E system has, by necessity, developed into a comparatively 
well organised, well documented and effective, if complex, system that meets the 
requirements of the U.S. Defence acquisition process. 
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IV. RAN SHIP OPERATIONAL TEST & EVALUATION 



The recent disestablishment of the RAN’s independent Operational Test and 
Evaluation authority, the RAN Trials and Assessing Unit (RANTAU), and its 
subsequent integration with Maritime Headquarters (MHQ), marks a continuing 
shift in RAN OT&E policy. This chapter describes the RAN OT&E system, its 
relationship with the other categories of T&E and its role in the RAN ship 
acceptance process. As RAN T&E documentation does not yet reflect the 
disbanding of RANTAU, the OT&E system is described as it was prior to May 
1992, and then the new organisation will be introduced. 

A. AUSTRALIAN DEFENCE ACQUISITION PROCESS 

Acquisition of major capital equipment for the Australian Defence Force ( ADF) 
is managed by the Capital Procurement Organisation (CPO) within the Department 
of Defence. The defence system life cycle consists of the following phases: 

• Conceptual 

• Demonstration and Evaluation 

• Full Scale Engineering Development 

• Production 

• In Service 

Similar, but not identical, to the U.S, system, these phases are generally 
separated by the following milestones: 

• Program Initiation 

• Full Scale Engineering Development 

• Production Approval 
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Preliminary Acceptance (PA)/ Delivery 
Acceptance into Naval Service (AINS) 
Operational Acceptance (OA) 






FULLSCALE 

PROGRAM ENGINEERING PRODUCTION PRELIM AINS OA 

INITIATION DEVELOPMENT APPROVAL ACCEPT 

APPROVAL 




Figure 9: Australian Acquisition Process for Major Programs 



1. Development Vs. Procurement 

The completion of all acquisition phases applies only to indigenously 
developed systems. Systems procured overseas generally undergo only those phases 
from Milestone 3 onwards. Due to the RAN’s small size and limited requirements, 
it is not practical or achievable to develop all required systems locally. In many 
cases, systems with the fundamental capabilities required by the ADF are available 
overseas, however, some of these systems require additional development, 
adaptation or integration to meet Australian specific requirements. Many of these 
systems require only minimal adaptation (e.g., in their communications fit), while 
others are more extensive (e.g., ANZAC class frigates and COLLINS class 
submarines). This leads to different types of system acquisition programs. 
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2. Types of Acquisitions 

The following are the principal ways Australia acquires defence 
capabilities [Ref. 6:p. 86]: 

a. Indigenous Design and Development 

Local research and development can be undertaken to meet special 
local requirements, or as consequence of research undertaken to maintain the 
technology base. The new system is subject to all phases of the acquisition process. 

b. Local Development and Production 

Naval Adaptation programs in which an overseas design is purchased, 
but then undergoes modifications to meet specific RAN requirements. Some 
engineering development work may be needed. 

c. Local Production of Overseas Designs 

In some cases, the requirements of the ADF can be met most cheaply 
and expeditiously by existing overseas systems where there is no economic or 
technical prospect of a local competitor. The system may be built locally, where it 
would be subject to Production Approval. 

d. Import of Overseas Equipment 

Non development programs, where systems are purchased “off-the- 
shelf’ from a foreign source, without modification, are subject to the acceptance 
milestones only. 

Each of these types of acquisition differ in their technical and operational 
risk and hence T&E requirements. In general, Australia has a policy of buying low 
risk, operationally proven systems, which may be modified to meet local 
requirements. 
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B. AUSTRALIAN DEFENCE T&E DEFINITIONS 

The categories of ship T&E used in the Australian DoD have been derived from 
the U.S. DoD instructions, but interpreted to accord with Australian requirements 
[Ref. 38:p. 14- 1J. There is a hierarchy of documents pertaining to T&E within the 
DoD and RAN. 

1. DoD T&E Definitions 

At the DoD level, the Australian Capital Equipment Procurement Manual 
(CEPMAN 1) [Ref. 38] defines the types of T&E applicable to the Australian DoD 
as follows: 

a. Development Test and Evaluation (DT&E) 

DT&E assists the development specification, design and procurement 
process and to verify the attainment of development specifications. 

b. Production Acceptance Test and Evaluation (PAT&E) 

PAT&E is to assist the development of a procurement specification and 
to verify compliance with its requirements. It is generally associated with a 
production activity and normally forms the basis of qualified acceptance of the 
equipment. 

c. Operational T est and Evaluation ( OT &E) 

OT&E is to assist the development of an operational capability 
specification and to measure compliance with its requirements. CEPMAN 1 also 
defines a sub-grouping within the OT&E classification - the Operational Evaluation 
(OPEVAL). An OPEVAL covers tests and evaluation on production representative 
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baseline equipment using the maintenance and support personnel and equipment for 
normal operational use and aims to: 

• Demonstrate operational effectiveness and suitability 

• Provide data to assist in the development of tactical aspects of the equipment. 

• Verify data, handbooks and documentation covering the operation of the 
system. 

2. RAN T&E Definitions 

These DoD definitions are further refined by the RAN. The Defence 
Instruction for RAN Test and Evaluation Policy [Ref. 44] defines the types of T&E 
as follows: 

a. Development Test and Evaluation (DT&E) 

DT&E denotes that T&E conducted in order to demonstrate the 
progressive achievement of design requirements and to show that the article will 
meet its technical specifications. DT&E is divided into three phases as shown in 
Table 7. 



TABLE 7: PHASES OF DT&E [Ref. 44:p. 2] 



T&E Type 


Description 


DT-1 


Validation of design concept 


DT-2 


Proving design 


DT-3 


Demonstration that production meets required technical 
characteristics 



78 



b. Production Acceptance Test and Evaluation (PAT&E) 

PAT&E denotes that T&E conducted by or on behalf of the 
procurement agency to determine whether contracted provisions have been 
satisfied. PAT&E is divided into two sections with a total of eight phases as shown 
in Table 8. 



TABLE 8: PHASES OF PAT&E [Ref. 44:p. 2] 



PAT&E Part 


T&E Type 


Description 


Part 1 


PAT-0 


Design and engineering development tests 




PAT-1 


Production and bum-in tests 




PAT-2 


Environmental qualification tests 




PAT-3 


System development tests 




PAT-4 


Harbour testing 




PAT-5 


Sea testing 


Part 2 


PAT-6 


Operational and Qualification Trials 
(conducted with OT-3) 




PAT-7 


Follow on Operational T&E 
(conducted with OT4) 



c. Operational Test and Evaluation ( OT&E) 

OT&E is that T&E conducted to estimate the operational effectiveness 
and suitability by an authority independent of the development and production 
agencies. OT&E is divided into five phases as shown in Table 9. Of note is that 
Certification/Qualification trials are included in OT-3. These trials consist of 
Communications, Emitter and other individual system certifications. These trials 
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also include a five to seven week Combat System Ship Qualification Trial (CSSQT) 
that: 

• Demonstrates the maintainability and operability of those equipments/ 
systems that are included in the CSSQT program through accomplishment of 
equipment/system Planned Maintenance System (PMS), aircraft/balloon 
tracking exercises, practice missile firings and culminating in live firing 
exercises. 

• Provides training and familiarisation to ship personnel in the maintenance 
and operation of the installed equipments/systems that are included in the 
CSSQT program. 

• Identifies design problems in the equipment/system installed. 

• Identify any deficiencies that may exist in the CSSQT support elements, i.e., 
documentation, logistics, test equipment or training. 



TABLE 9: PHASES OF OT&E [Ref. 44:p. 2] 



T&E Type 


Description 


OT-1 


Operational assessment of the development proposal 


OT-2 


Demonstration of achievement of program requirements for 
operational effectiveness and suitability of a prototype 
(OPEVAL) to support proceeding to full production 


OT-3 


Demonstration of achievement of program requirements for 
operational effectiveness and suitability on production of ship/ 
aircraft/system, normally in independent operations using normal 
Fleet personnel. Certification/Qualification TYials include 
limited reliability, maintainability, availability and logistic 
supportability assessments. 


OT-4 


Demonstration of achievement of program requirements for 
operational effectiveness and operational suitability on 
production of ship/aircraft/system using normal Fleet personnel 
in a multi-force, multi-threat environment. Includes detailed 
reliability, maintainability, availability and logistic supportability 
assessments. 


OT-5 


Follow-on OT for assessment after modernisations, new 
applications or defect rectifications after OT-4 
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C. T&E IN THE ACQUISITION PROCESS 



1. The Need for T&E 

The most recent Australian Government “White Paper” on defence policy 
[Ref. 6:p. 70] states: 

We need to be able to determine the performance in our own environment of 
equipment of both overseas and local origin and to modify and adapt overseas 
equipment as necessary to improve its performance in our likely theatres of 
military operations. 

Also, the guidance for Australian defence acquisition [Ref. 38:p. 14-1] 
recognises that: 

The evaluation of military weapons systems and their individual component 
systems is essential to this policy. Analysis of the results of testing performed 
as part of the Test and Evaluation (T&E) of a capital equipment project may 
assist in the identification of problem areas, allow timely corrective action to 
be taken, and reduce the element of risk in major decisions. 

So the need for local T&E is recognised by the Australian Government and 
the DoD, for both indigenous and overseas systems. The responsibility for T&E of 
a system rests with organisations with responsibility for the design approval, 
certification or procurement of equipment, who have the authority and responsibility 
to conduct (or require the conduct of) T&E. All project requirements are coordinated 
by the Project Manager, who considers whether or not T&E can be conducted within 
available project resources. [Ref. 38:p. 14-2] However, the importance of OT&E in 
particular, is not specifically addressed in any of these T&E policy documents. 

2. OT&E Contributions at Major Milestones 

OT&E in the RAN may be conducted during each phase in the acquisition 
process, but it is not essential to a local system meeting a development or production 
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milestone, or the local production or purchase decision of an overseas system. It is 
however, required before a system achieves Acceptance into Naval Service or 
Operational Acceptance. The phasing of T&E during the acquisition process is 
detailed in Figure 10. 
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Figure 10: Test & Evaluation in the Acquisition Process [Ref. 44:p. B-1J 



The contribution of OT&E to the acquisition milestones includes: 

a. OT & E prior to Program Initiation 

No role is documented for OT&E in the Conceptual Phase. 

b. OT &E prior to Engineering Development Approval 

During the Demonstration and Evaluation Phase, an operational 
assessment of the development proposal (OT-1) may be conducted. The results of 
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which may contribute to the decision to undertake Full Scale Engineering 
Development. 

c. OT&E prior to Production Approval 

Towards the end of the Full Scale Engineering Development Phase, a 
demonstration of achievement of program requirements for operational 
effectiveness (OT-2) and suitability of a prototype (OPEVAL) to support 
proceeding to full production. This phase may include system/software assessment 
at a Land Based Test Site [Ref. 44:p. 2], The results of OT-2, if conducted, may be 
used to support a local production or purchase decision. 

d. OT &E prior to Preliminary Acceptance 

No role is documented for OT&E in the Production Phase. 

e. OT &E prior to Acceptance into Naval Service 

Although Figure 10, indicates an OT-3 phase prior to AINS, no specific 
requirement is documented in RAN policy for the conduct of OT-3 prior to AINS. 
The prerequisites for AINS detailed in Reference 41 p 15.1 1, are described in terms 
of PAT&E Part 2, and make no mention of OT&E. At the working level, however, 
the RANTAU Standard Operating Procedure for OT&E [Ref. 53] includes the 
conduct of OT-3 prior to AINS, and the results of OT-3 have usually been 
considered in the AINS decision. 

f OT &E prior to Operational Acceptance 

For first of class ships it will not normally be possible to trial and 
evaluate all aspects of the operational performance and maintainability prior to 
AINS. Therefore, the first of class usually undergo follow-on OT&E (FOT&E) after 
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AINS, leading to Operational Acceptance [Ref. 44:p. A-l]. This OT-4 Phase 
consists of: 

• Evaluating the ship as part of a force during an exercise, in order to develop 
standard operating procedures and tactics, and to explore the limits of 
capability, etc. 

• Evaluating the maintainability and stores support for equipments and systems 
to determine whether or not the complement t, stores allowances, 
documentation are adequate. [Ref. 41:p. 15.14] 

g. OT &E Contributions post Operational Acceptance 

OT-5 is follow-on OT&E for assessment after modernisations, new 
applications or defect rectifications after OT-4 [Ref. 44:p. 3]. 

The OT&E contributions to the Acquisition Milestones are by no means 
definitive. There is no documented requirement for the conduct of OT-1 and/or OT- 
2 prior to a production or purchase decision of major acquisitions. The decision to 
conduct OT&E being the responsibility of the Sponsor, Project Manager and the 
Defence Acquisition Committees. RAN T&E policy documentation almost 
exclusively focuses on the post production acceptance phases. Even during the 
acceptance phase, however, the requirement for OT-3 is clouded in definition 
differences between policy documents. The RAN appears to have its focus on 
Production Acceptance Testing, possibly because most systems procured in the past, 
were already in service with a large parent Navy. Tactical development was not 
necessary because the tactics also were acquired with the system. 

3. Production Acceptance Test and Evaluation 

As defined previously, Production Acceptance Test and Evaluation 
(PAT&E) is the testing conducted to demonstrate that systems meet contract 
specifications and requirements: also that items/systems are properly installed and 
operable onboard the ship. The objective of PAT&E is to confirm that the contractor 
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has met contractual obligations, and hence the Commonwealth may correctly take 
delivery of the ship. [Ref. 41 :p. 14-2] Much emphasis is given within the RAN to 
PAT&E, known under various terms including Inspections, Tests and Trials (ITT), 
Harbour Acceptance Trials (HATs) and Sea Acceptance Trials(SATs). In recent years 
the RAN has adopted the U.S. Total Ship Test Program (TSTP) concept, initially 
with the modernisation of the DDGs, and now in the testing of the locally produced 
FFG-7 class, under the Australian Frigate Project. The adoption of the TSTP by the 
RAN included the redefinition of the test phases. Where the U.S. system included 
only PAT&E, the RAN has included elements of DT&E and OT&E, in addition to 
PAT&E, and gave them PAT designations. 

cl Integrated T est Package 

Under the TSTP, the program and content of the formal ITT required 
by the contract is usually designated the Integrated Test Package (ITP). Although in 
the Australian system, Government testing is also included. The categories of ITT 
testing are listed in Table 10. 



TABLE 10: CATEGORIES OF ITT [Ref. 41 :p. 14.3] 



PAT&E 


Category 


T&E Type 


Description 


Part 1 


0 


PAT-0 


Design and engineering development tests 




1 


PAT-1 


Production and bum-in tests 




2 


PAT-2 


Environmental qualification tests 




3 


PAT-3 


System development tests 




4 


PAT-4 


Harbour testing 




5 


PAT-5 


Sea testing 


Part 2 


6 


PAT-6 


Ship Operational and Qualification Trials 




7 


PAT-7 


Follow on Operational T&E 
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b. Overlap of PAT&E and OT &E 

PAT&E Part 2 covers the period from Preliminary Acceptance to the 
AINS and equates to PAT-6. The Project Director has responsibility for the program 
and its funding and RANTAU is responsible for the detailed schedule of events and 
the provision of test and trials resources. Testing includes: 

• Ship’s staff familiarization, shakedown and safety drills 

• Installation and testing of specialised equipment 

• Tempest test 

• Harbour trials of weapons, communication and combat data systems, etc. 

• Ship Qualification Trials 

• Degaussing and Noise Ranging 

However, some of these testing activities are also included as part of 
OT-3, and both PAT-6 and OT-3 are conducted by the same authority (RANTAU), 
during the same period, and using the same T&E personnel. Similarly, the PAT-7 
phase equates to OT-4. 

D. RAN OT&E ORGANISATION 

The RAN community responsible for planning, programming, budgeting and 
evaluating OT&E forms an integral part of the more general T&E organisation. 
OT&E in the RAN is not treated as being special or unique, and no organisation 
deals solely with OT&E. The RAN organisational structure has changed over the 
past few years which has impacted on the structure of the OT&E community. 

1. OT&E Organisation Elements 

The functional elements of an OT&E organisational structure include: 

• Policy Maker and Overseer 

• Sponsor 
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• Developing Agency 

• Tester and Evaluator 

• User 

• Coordinator 

a. Policy Maker and Overseer. 

There is no clear OT&E policy maker within the Australian DoD or the 
RAN. The Chief of Capital Procurement within DoD has a role in the formulation 
of capital equipment procurement policy for the guidance of Service Materiel 
Divisions. The CPO manual [Ref. 38] addresses T&E, but more in the form of 
guidance rather than policy. The Director General of Naval Warfare (DGNW) in the 
Deputy Chief of Naval Staff (DCNS) Division, drafted the RAN Test and Evaluation 
Policy Defence Instruction [Ref. 44]. DGNW however, is now defunct with the 
result that Test and Evaluation policy for Navy is presently in a state of flux, with 
no office having assumed the responsibility [Ref. 50]. 

b. Sponsor 

The sponsor is responsible for monitoring operational requirements and 
warfare concepts, the development of concepts for operations for naval warfare 
systems, and the raising of staff targets and requirements as required. He/She is also 
responsible for the subsequent trials and continuing overview to ensure that the 
equipment meets the requirement. Originally the responsibility of DGNW, this 
function for major acquisitions was assumed by the Director General Force 
Development (Sea) in Headquarters, Australian Defence Force (HQADF). This 
centralised force development and user requirements organisation produces the 
Naval Capability Proposals (NCP). The decision to subject an acquisition to T&E is 
initially decided by the sponsor and detailed in the capability proposal [Ref. 44:p. 1] 
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c. Developing Agency 

When a project is approved, a project office is formed within the 
Materiel Division responsible for acquisition of the capital equipment to meet the 
approved NCP. The nature and extent of the T&E to be conducted will be decided 
by the Project Director in consultation with the Design Approval Authority and 
RANTAU and detailed in the TEMP [Ref. 44:p. 1], The Project Manager, in 
consultation with operational, technical and maintenance authorities, is to fully 
investigate the necessity for, and likely scope of. Defence T&E [Ref. 38:p. 14-5]. 

(L Operational Tester and Evaluator 

The RAN Trials and Assessing Unit (RANTAU) was the RAN’s 
OT&E authority with a mission to provide an independent evaluation, audit and 
training service in response to current and future Maritime Force requirements. 
Located in North Sydney, near the main Fleet Base, the trials related functions of 
RANTAU included[Ref. 43:p. 1]: 

• Conduct of operational effectiveness and operational suitability trials on 
ships and facilities under construction, modernisation, conversion or 
extended refit in support of AINS. 

• Conduct test and evaluations, investigations and inspections of selected 
equipment when tasked. 

• Audit and witness selected Harbour Acceptance Trials (HATS), Sea 
Acceptance Trials (SATS) and Test Procedures which have operational 
implications for ships and facilities (now known as PAT 4 and 5). 

• Conduct Ship Qualification Trials. 

RANTAU, although the OT&E authority, also conducted PAT&E on 
behalf of Project Directors and/or the Maritime Commander. With the 
disestablishment of RANTAU in May 1992, the trials functions were transferred to 
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the Commander, Test and Evaluation (CTE) under the Chief Staff Officer 
(Engineering) in the Maritime Command. 

e. User 

The Maritime Commander, and the individual ships’ Commanding 
Officers and crews are the ultimate users of a ship, RAN T&E documentation, 
however, does not address their involvement in T&E. The Commanding Officer of 
a ship under test usually provides RANTAU with his opinions on the ship's 
performance at the conclusion of a trials period. 

/ Coordinator 

The Director of Naval User Requirements (DNUR), under DGNW, 
once acted as the Navy Office representative for RANTAU within Navy Office, 
Canberra. He provided a coordination/liaison role between RANTAU and the 
Materiel Division, by attending project meetings and highlighting T&E 
requirements. 

2. Changes to the RAN OT&E Organisation 

The OT&E organisation within the RAN has changed dramatically over the 
past few years. Figure 11 illustrates the OT&E organisation in 1987. The Director, 
RANTAU (of Captain rank) was directly responsible to the Assistant Chief of Naval 
Staff - Development (ACDEV-N). Also under ACDEV-N, the Director General 
Naval Warfare (DGNW) sponsored all major acquisitions, and gave T&E policy 
direction. Under DGNW, the Director of Naval User Requirements (DNUR), 
provided RANTAU representation and performed an OT&E coordination role 
within Navy Office. Thus the OT&E authority was responsible to the developer of 
the user requirement, and was independent of the developing and production 
agencies. 
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Figure 11: RAN OT&E Organisation circa. 1987 



By 1990, the structure outlined in Figure 12 had developed. With the 
centralisation of military policy development and resource planning, the 
Development Division within the RAN was disbanded. DGNW, with reduced user 
requirement responsibilities, moved under the Deputy Chief of Naval Staff (DCNS), 
The emerging HQADF now became the Sponsors for major acquisitions. The 
Director General Force Development (Sea) (DGFD(SEA)) having particular 
responsibility for “Sea” Capability Proposals [Ref. 54]. DGNW, however, retained 
responsibility for T&E policy within the RAN. RANTAU was also transferred under 
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DCNS. As the sponsorship of major acquisitions was now held by HQADF, the 
OT&E authority was now no longer responsible to the developer of the user 
requirement. DNUR was dissolved and the OT&E authority lost its representation 
in Navy Office. 




Figure 12: RAN OT&E Organisation circa. 1990 



By the end of May 1992 (Figure 13), RANTAU was disestablished and its 
trials functions were transferred to the Maritime Commander as the Commander 
Test and Evaluation (CTE) under the Chief Staff Officer (Engineering) (CSO(E)). 
DGNW was dissolved and his now reduced functions were assumed by the Director 
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Naval Warfare (DNW) under the newly created Director General Naval Policy and 
Warfare (DGNPW)fRef. 55] within the DCNS Division. 




Figure 13: RAN OT&E Organisation post May 1992 



So, as a result of reorganisations over the past five years, the RAN has no 
T&E, let alone OT&E, policy maker and overseer. It also has no OT&E coordinator 
within Navy Office and the RAN’s OT&E authority is buried within the in-service 
engineering management area of the Maritime Command. The OT&E authority, 
however, remains independent from the developing and production agencies. 
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E. RAN OT&E POLICY 

RAN OT&E policy is embedded in the more general T&E policy. 

1. Policy Documentation 

The following documents outline the RAN T&E policy: 

a. Capital Equipment Procurement Man ual ( CEPMAN 1 ) 

Australian DoD T&E policy has its basis in the Capital Equipment 

Procurement Manual (CEPMAN l)[Ref. 38], the aim of which is to assist project 
managers in assessing the need for T&E in a project and to provide guidelines for 
its conduct. The document provides broad T&E policy and, other than defining 
OT&E, little policy is provided as to its conduct. 

b. DI(N) LOG 82-1, RAN Test and Evaluation Policy 

The top level T&E policy document for the RAN is the Defence 
Instruction (Navy) on RAN Test and Evaluation Policy [Ref. 44]. Its purpose is to 
issue policy for the conduct of T&E in capital procurement and modernisation 
projects in the RAN. 

c. ABR1921 

ABR 1921 [Ref. 41], contains T&E and acceptance policy applicable to 
ships building, undergoing modernisation, conversion or extended refit. Although it 
refers to OT&E under PAT&E Part 2, it does not address OT&E specifically. 

2. T&E Decisions 

RAN T&E policy [Ref. 44:p. 2] states that the decision to subject an 
acquisition to T&E is initially decided by the Sponsor and detailed in the Capability 
Proposal. The nature and extent of the T&E is then decided by the Project Director 
in consultation with the Design Approval Authority and RANTAU. Any conflict 
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over the requirement, nature and/or extent of the proposed T&E is resolved by 
DCNS and ACMAT-N for operational and design matters respectively. However, in 
practice, the nature and extent of T&E is broadly decided by the Sponsor, and the 
Project Director is responsible after consultation with the DAA and the Trials 
Authority [Ref. 45]. 

3. Independence. 

The independence of a testing authority is recognised in general terms in 
CEPMAN 1 [Ref. 38] which recognises that independent verification and validation 
(IV&V) is an important means of providing early detection of problems, and that it 
can have value and credibility. However, it implies that “independence'’ means 
independent from the Defence Department, i.e., contractors. With respect to OT&E, 
CEPMAN 1 compromises the independence of an OT&E authority by stating that 
the Design Approval Authority (DAA), which is part of the Developing Agency, is 
responsible for the assessment of the acceptability of the product design as a basis 
for Acceptance into Service. It provides for the DAA being involved in offering 
support to trials establishments (as required) during OT&E in order to provide DAA 
advice to the Project Manager on the suitability of the product for Acceptance into 
Service. [Ref. 38:p. 14-7] This appears to be more applicable to services other than 
the RAN, as the DAA within Navy does not advise the PM on the suitability for 
AINS. 

The RAN T&E Policy DI(N) [Ref. 44: p. 3] is more clear when it states 
OT&E is to be conducted progressively by an authority independent of the 
development and production agencies (usually RANTAU). With the demise of 
RANTAU and the transfer of OT&E responsibilities to CTE on the staff of the 
Maritime Commander, independence from the development and production 
agencies is maintained. 
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4. Use of Contractors 

Although the Australian definitions of OT-3 and OT-4 (Table 9) includes the 
requirement that the system should be demonstrated “using normal Fleet 
Personnel”, the documentation makes no policy statement regarding contractor 
personnel involvement in system operation or maintenance, during OT&E. It has 
been RAN practice (e.g., in the DDG Modernisation Project) to have contractors 
onboard resolving technical problems during the SQT period, which is included as 
part of OT-3 in the RAN. 

5. T&E Planning 

The RAN uses the Test and Evaluation Master Plan (TEMP) as the single, 
executive, long range planning document for T&E in a project. The TEMP is to be 
raised in draft form as early as practicable. Based on the USN format, it is the 
ultimate responsibility of the project sponsor because it is the primary tool to ensure 
the user requirement is met. However, during the period when there is a full time 
Project Office, TEMP coordination is normally carried out by the T&E Manager in 
the Project Office. RANTAU (now CTE) as the OT&E authority provides the OT&E 
input. The TEMP is endorsed by all affected organisations and is released jointly by 
the Deputy Chief of Naval Staff (DCNS), and the Assistant Chief of Naval Staff- 
Materiel (ACMAT-N), as the Developing Agency. If any issues remain unresolved 
at this level, the TEMP will be resolved by the Chief of Naval Staff (CNS). [Ref. 
44:pp. 4-5] The authority for DCNS to release the TEMP was based on his role as 
Project Sponsor. With HQADF now assuming this role, the DCNS function in 
TEMP development is unclear. 
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6. Combined DT&E and OT&E 



Although not specifically addressed as such, combined DT&E, (PAT&E) 
and OT&E is implied as being condoned and almost mandatory within the RAN. 
The T&E DI(N) [Ref. 44:p. 3] states that OT-3 and OT-4 will incorporate the 
requirements of PAT-6 and PAT-7 Inspection Tests and Trials as described in ABR 
1921. It also states that OT-3 will be conducted during Ship Qualification Trials. No 
limitations or guidance as to the possible hazards of this approach are addressed. 

7. Modeling & Simulation 

The use of models and simulation (M&S) in OT&E are not addressed by 
the RAN documentation, and their application to OT&E has been limited. 
Simulation and stimulation were used extensively during DT&E by both the 
Submarine Warfare Systems Centre (SWSC) and the Combat Data System Centre 
(CDSC) for the Submarine Weapons Update Program (SWUP) and DDG 
Modernisation respectively. However, Major combatant OT&E applications of 
M&S within the RAN have been limited to the DDG Modernisation Project. M&S 
developed by the Maritime Systems Division of the Defence Science and 
Technology Organisation were used by RANTAU for the evaluation of the upgraded 
combat and weapons systems in the Modernised DDGs. 

8. Foreign Weapons Evaluation 

Despite the most recent Australian Government “White Paper*’ on Defence 
Policy [Ref. 6:p. 70] requiring the need to be able to determine the performance in 
the Australian environment of equipment of both overseas and local origin, no DoD 
or RAN T&E policy specifically addresses the evaluation of foreign systems. 
Although, almost all of the systems procured by the RAN in recent years are 
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“foreign” in origin, the RAN documentation focuses on the T&E process for the 
progressive evaluation of locally developed systems. 

9. OT&E Funding 

OT&E is funded separately within a project and is estimated and bid for by 
RANTAU to Project Directors. [Ref. 44:p. 6] 

10. Land Based Test Sites 

The guidance for OT&E using land based test sites includes a statement that 
system centres and simulators will be employed for early stages of OT&E if 
available [Ref. 44:p. 4], and that OT-2 may be conducted at a land-based test site 
[Ref. 44: p. 3]. 

11. Ship OT&E 

The RAN recognises that some major acquisition projects will be treated 
differently to others in so far as protracted construction time usually precludes T&E 
of a prototype before the decision to proceed to production. The degree of technical 
risk will be assessed in each case and, where considered sufficient, the decision 
taken to use land-based test sites and/or prove a particular system in another 
platform. When this occurs, DT-2 and OT-2 shall be conducted at the test site and 
the “whole ship” evaluation undertaken as soon as practicable after delivery. 

F. RAN SHIP ACCEPTANCE 

1. Policy 

The decision to accept a new or modified ship into the RAN is vested in the 
Chief of Naval Staff (CNS). An Acceptance Board is established to provide an 
assessment of the ship and submit recommendations upon which CNS can base his 
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acceptance decision. The assessment of a ship leading to an acceptance decision has 
two distinct components [Ref. 41 :p. 15.7]: 

• Determining where the ship as built and tested does not meet the Agreed Ship 
Characteristics, contracted performance specifications and approved support 
standards. 

• Deciding whether those Agreed Ship Characteristics, performance 
specifications and support standards accord with current policies standards 
and practices. 

The Acceptance Board (AB) is an independent body established by CNS to 
make recommendations for CNS's acceptance into RAN service of new 
construction or modernised ships, submarines, aircraft, and, where directed, new 
systems, installations and equipment. Usually established for each new ship type 
and only for first of class, a Board is composed of a President, a Vice President (VP), 
and additional members selected from appropriate specialist areas. The President, 
VP and Board members serve on the Board part time. The Board is administered by 
the Director, RANTAU (now CTE) who is an ex officio member, assisted by a full 
time Secretary. [Ref. 42:pp. 1-3] The AB's sole function is to advise CNS on 
acceptance matters, and it is essentially a board of review. 

The RAN divides acceptance into three phases: 

• Preliminary Acceptance 

• Acceptance into Naval Service 

• Operational Acceptance 

2. Definitions 

The “Delivery Commissioning and Acceptance” chapter of Reference 41 
provides the definitions for the acceptance process: 

• “Delivery” occurs when the ship is contractually delivered from the 
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contractor’s ownership to the Government. 

• ‘"Preliminary Acceptance” is an assessment by the RAN to determine if the 
ship has reached both material and support status sufficient to safely and 
effectively proceed with the Navy trials and evaluation program. It usually is 
concurrent with Delivery. 

• “Acceptance into Naval Service” (AINS) signifies the acceptance of the ship 
as a fully operational unit of the Fleet, based on operational performance and 
support levels obtained at that time. 

• “Operational Acceptance” is carried out only on the first of class to 
supplement the AINS assessment using additional operational and support 
experience gained with the Fleet to refine the vessel and Naval 
Requirements. 

3. Responsibilities 

Delivery of the ship to the Government is a contractual matter and is 
managed by the Project Director. Once delivered, the process is one of 'offer and 
accept’ whereby the ship is offered by the Project Director to CNS for acceptance. 
RANTAU’s responsibility is to conduct the Acceptance Trials and report the results. 
The Acceptance Board does not conduct tests, trials or inspections, but may witness 
them if required. [Ref. 42] The Acceptance Board roles are as follows: 

• Be satisfied that a comprehensive and properly supervised series of 
inspections, tests, trials and evaluations have been carried out to the point 
where assessments can be made on current suitability for operational service 
on the basis of the approved current requirements. 

• Assess the ship, aircraft or equipment, by examining reports of materiel 
inspections, operational trials and evaluations, attending trials as required 
and calling for such other tests or evaluations as necessary. 

• Be satisfied that current, relevant and approved documents when 
promulgated describe the required operational characteristics against which 
the item for acceptance can be tested and evaluated for operational 
suitability, operational effectiveness and supportability in service. 
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4. Procedure 



The completion process for a warship entering service is outlined in Figure 
14. This process may be divided into two distinct parts: 

• Those events leading to and culminating in delivery / preliminary acceptance. 

• Subsequent activities leading to AINS and Operational Acceptance (if first of 
Class) 




Figure 14: RAN Ship Acceptance Schedule and Milestones 



a. Delivery / Preliminary Acceptance 

Delivery of the ship to the Government is a contractual matter and is the 
prerogative of the Project Director acting on behalf of ACMAT-N and the 
Commonwealth of Australia [Ref. 41 :p. 15.3]. Immediately prior to delivery, the AB 
will advise CNS whether the ship has reached a materiel and support status suitable 
for Preliminary Acceptance. Preliminary Acceptance assures CNS that the vessel is 
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considered safe for a Naval crew to take control of the vessel from the contractor and 
conduct outstanding RAN trials of equipment at sea or undertake specific trials for 
AINS assessment. [Ref. 41 :p. 15.10] 

b. Acceptance into Naval Service 

Acceptance into Naval Service is essentially the acceptance/transfer of 
the ship from the acquisition part of the Navy (ACMAT-N) to the inservice 
operations and support areas, as having met the user requirement. The AINS 
assessment is carried out on completion of the Post Delivery Availability (PDA) 
Trials Period, at which point the ship should be in a materiel state ready for 
operational service. Post delivery trials leading to AINS include the SQT period, 
with harbour and sea elements. While the PD is responsible for the PAT&E 
program, RANTAU is responsible for the detailed planning and witnessing of the 
trials program consulting with the PD. During these trials, the PD may provide a full 
time trials coordinator onboard to assist the Commanding Officer and the other trials 
authorities. [Ref. 41 :pp. 15.10 - 15.12] On completion of the trials period, the AB 
will advise CNS whether the materiel and support state of the ship is sufficient for 
it to enter service as an operational unit of the Fleet, and hence it can be accepted 
into naval service [Ref. 42:p. 4]. 

c. Operational Acceptance 

For a first of class ship it will normally not be possible to trial and 
evaluate all aspects of the operational performance and maintainability prior to 
AINS without the duration of the PAT&E Part 2 program becoming excessive and 
because certain aspects can only be evaluated in the operational environment. This 
follow-on T&E period is conducted following AINS and is compiled by RANTAU 
in consultation with the Naval and Maritime Headquarters Staff. It comprises a 
series of trials and evaluations to assess operational effectiveness and operational 
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suitability. [Ref. 41 :p. 15.14] Typical examples of these activities in the FOT&E 
program are: 

• the evaluation of the ship as part of a force during an exercise, in order to 
develop standard operating procedures and tactics, and to explore the limits 
of capability, etc. 

• evaluation of the maintainability and stores support for equipments and 
systems, to determine whether or not the complement, stores allowances, 
documentation, etc. are adequate. 

Typically, the Follow-on T&E program will last from 12-24 months, 
depending on the class of ship. At the end of this period RANTAU compiles a report 
in consultation with Maritime Headquarters Staff, the ship and other authorities as 
appropriate, indicating: 

• The results of the trials and evaluations 

• Whether any changes are recommended to the proposals by the PD for 
correcting deficiencies which were extant at the end of PAT&E Part 2 trials. 

• What additional items, if any should be added to the list of deficiencies and 
what action is proposed to overcome them. 

The AB assesses the results and provides the overview to ensure the 
program is comprehensive. This assessment serves to confirm or modify the 
assessment at AINS. 



G. RAN OT&E PROCEDURES 

Procedures are the methodology by which policy is carried out. This section 
discusses the procedures adopted by the RAN in implementing OT&E policy. 
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1. User Requirement to the Acceptance Management Plan. 

The development of the user requirement, and its subsequent progression 
through to the OT&E test documentation (Figure 15) is through the following 
documents: 

• Operational Concept Paper (OCP) 

• Defence Force Capability Proposal (DFCP) 

• Required Operational Characteristics (ROC) 

• Approved Ship Characteristics(ASC) 

• Test and Evaluation Master Plan (TEMP) 

• Acceptance Management Plan (AMP) 




Figure 15: Progressive Definition of OT&E Documentation 
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a. Operational Concept Paper ( OCP) 

The strategic basis of papers (DOA-87 [Ref. 6] and ASP-90 [Ref. 5]) 
provide endorsed Government policy guidance for force structure planning. These 
documents identify ADF roles and establish broad priorities for capability 
development. From this guidance. Operational Concept Papers are produced by 
HQADF to identify the way ahead for specific force structure issues. [Ref. 54:p. A-4] 

b. Defence Force Capability Proposal (DFCP) 

Against the background of an OCP, a Defence Force Capability 
Proposal (DFCP) is developed by HQADF. The DFCP is the document against 
which specifications are produced and ultimately the proposed solution is evaluated 
as being suitable for service use. [Ref. 54:p. A-5] 

c. Required Operational Characteristics (ROC) 

To meet the overall requirements of the DFCP, more specific 
requirements may be described in the Required Operational Characteristics. 

d. Agreed Ship Characteristics (ASC) 

More detailed requirements, usually for ship programs, are developed 
into the Agreed Ship Characteristics (sometimes termed Approved Ships 
Characteristics). Other subordinate performance documentation may be developed 
also. 

e. T est and E valuation Master Plan (TEMP) 

For a new ship design, the standards against which the ship is assessed 
are those derived from the then approved DFCP, ship characteristics and policies. 
These standards are developed into a TEMP which details the responsibilities for the 
specification of requirements, objectives, criteria and conduct of T&E, together with 
the schedule of activities, resources and key contractual dates. The most important 
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part of the TEMP is the statement of Critical Operational Issues (COI) which are 
either of high technical risk or vital to the continuing development process. These 
issues may not be dependent on thresholds, but should cover all areas that affect the 
systems capability to accomplish its mission in a combat related environment. [Ref. 
44: p. 5] 

f Acceptance Management Plan (AMP) 

The Trials Authority develops management plans for the trials it 
conducts. Where those trials lead to AINS, the management plan is termed the 
Acceptance Management Plan. The AMP takes the COIs from the TEMP and details 
the tests and trials necessary to evaluate them to support an AINS decision. 
Similarly a Follow on OT&E (FOTE) Management Plan may be produced detailing 
the trials for the OT-4 phase. The detailed trials plans are developed from these 
management plans. 

g. Summary 

As in the USN, the hierarchy of requirements documentation in the 
RAN provides the linkage from capability requirement to operational test program 
necessary to evaluate the final product against the user requirement. 

2. Assessment of Operational Effectiveness 

The RAN has found it difficult to define exactly what is required to assess 
operational effectiveness. The RANTAU guidance on the conduct of OT&E [Ref. 
53] recognises that the user requirement is the bench mark for determining the 
degree to which a product is effective. Also that operational effectiveness is best 
assessed by a performance demonstration by normal operating personnel in the 
normal or given environment. But otherwise provides very little guidance on the 
assessment of operational effectiveness. Techniques employed involve analysing 
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each COI and employing modified Ship Qualification Trial and Fleet Exercise 
Program techniques to evaluate them. Use has also been made of simulation, 
weapon analysis and exercise analysis. The lack of suitable range facilities has also 
limited the assessment of operational effectiveness. 

This lack of operational effectiveness assessment capability in the RAN is 
due to most systems being acquired from large parent Navies. Consequently, the 
RAN has seldom had the need to assess operational effectiveness for itself. The 
Submarine Project is acquiring a variety of ranges for DT&E applications, e.g., the 
measurement of signatures, manoeuvreing characteristics, weapon control 
characteristics, etc. These ranges may also be suitable to support OT&E. 

3. Assessment of Operational Suitability 

The assessment of operational suitability in the RAN has been 
controversial, particularly the assessment of supportability. This quote from the then 
Director of Naval Integrated Logistic Support Management [Ref. 40] in 1987 
highlights a particular viewpoint: 

While there are clearly visible yardsticks by which production or weapon 
system performance can be measured, the success of logistic planning is to an 
extent invisible. Shortcomings in the assessment and provision of support will 
only become evident due to operational failure. All that RANTAU or the Ship 
Acceptance Board can do is make a subjective judgement as to whether all 
necessary ILS elements have been addressed and ensure that an acceptable 
follow on logistic support and enhancement process has been set in place. 

Before 1987, the assessment of ILS, a subset of operational suitability, was 
rather subjective. The assessment of supportability, for example, consisted of a 
review of allowance lists by experienced technical specialists where potential 
deficiencies were highlighted. The ILS community were concerned that the onboard 
spares allowances developed by detailed logistic support analysis, were being 
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subject to a relatively simplistic, qualitative assessment. Similarly the assessment of 
Reliability Maintainability and Availability (RM&A) was superficial with only 
those defects noted during the operational assessment period (from five to seven 
weeks) being analysed. A new approach was taken in 1987 to the assessment of 
operational suitability, and to ILS in particular. The assessment process has 
continued to be refined with the areas in Table 1 1 now assessed: 



TABLE 11: SUITABILITY ISSUES FOR ASSESSMENT 
Suitability Issues 

Availability 

Reliability 

Maintainability 

Safety 

Human Factors 
Interoperability 
Compatibility 

Integrated Logistic Support including: 

Maintenance Planning 

Supply Support 

Support and Test Equipment 

Technical Data and Documentation 

Manpower and Personnel 

Training and Training Devices 

Facilities 

Packaging, Handling, Storage and Transport 
Computer Support 
Configuration Control 



a. Suitability Assessment Procedure 

Safety is assessed by inspection and observation during system 
operation. Human factors, interoperability and compatibility are assessed by 
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observation during operational effectiveness testing. The assessment of ILS is 
conducted in two phases: 

( 1 ) ILS Structural Review. Conducted prior to operational testing, this 
review assesses if the ILS intended to provided will support the ILS policy of the 
Project. It consists primarily of a review of the ILS Plan and discussions with the 
functional areas handling the support. 

(2) ILS Operational Review. During the operational testing period, the 
ship's ILS related documentation and records are reviewed, maintainability 
demonstrations are performed and a supportability assessment is conducted. The 
supportability assessment still consists of a review of the allowance lists by 
experienced technicians, but now it also includes an analysis of spares usage and 
availability, from the time records are commenced to the end of the operational 
testing. 

RM&A is assessed by the collection and analysis of data during the OT- 
3 phase of operational testing to give an indication of performance prior to AINS, 
and then continued for 12 months during the OT-4 phase. The RAN is currently 
developing a routine in-service RM&A data collection and analysis system, 
however, until that is operational, the data is collected and analysed on an ‘as 
required’ basis by the Trials Authority. 

4. OT&E Reporting 

RANTAU coordinates the issue of trials reports which are issued on 
completion of each segment of OT&E e.g., o/c ASW phase of OT-3 etc. Afinal trials 
report is issued to the AB for inclusion in their AINS report. 
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5. OT&E Coordination 

A T&E Planning Group is formed prior to TEMP development under the 
leadership of the T&E manager for the project. Foundation members of the TEPG 
are the Project Director, Project Sponsor, the Design Approval Authority and 
RANTAU, with other parties seconded/called upon as required. [Ref. 44:p. 5] The 
TEPG analyses T&E requirements and estimates resources required to meet the 
requirements. 

6. OT&E Personnel Selection and Training 

Although OT&E is a specialised discipline with its own philosophy and 
methodology, the RAN has no training courses on OT&E, or on T&E in general. A 
number of officers have completed the USN Operational Test Directors Course in 
recent years which has improved the knowledge and understanding of OT&E within 
CTE’s organisation considerably. These courses have been arranged on an ad-hoc 
basis through overseas visit submissions rather than as pre-requisite courses for 
particular billets. No special selection criteria is involved for posting OT&E 
personnel or project T&E managers. Although some personnel may develop into 
OT&E specialists, no career path is consciously provided. 

H. RAN OT&E IMPLEMENTATION 

The implementation of OT&E policy within the RAN maybe demonstrated by 
reviewing examples of ship OT&E conducted to date, and that planned for ships due 
to enter service in the near future. 

1. DDG Modernisation Project 

The RAN has three DDGs built in the USA to a modified “Charles F. 
Adams” (DDG-2 class) design. These ships underwent a major modernisation and 
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refit over the period 1987-1991. The major modernisation occurred with the ships’ 
combat system, command and control system, sensors and weapons, with the aim of 
increasing the ‘supportability’ of the systems rather than increasing the ships’ 
capability [Ref. 39]. The upgrade was similar to that undertaken on a small number 
of USN DDGs. Most of the systems were imported and installed locally, however, 
the combat system operational program was developed in Australia. The risk of the 
project failing to meet operational requirements was low, but the RAN needed to 
quantify the operational effectiveness and operational suitability of the modernised 
DDG. 

OT&E consisted of an OT-3 assessment during a normal SQT period, prior 
to workup and the Operational Readiness Evaluation. This was followed by an OT- 
4 phase, termed a Combat System Evaluation (CSE), for the first of class. This OT- 
4 was event based and was completed over a 12 month period, integrated with the 
normal ships program. Procedures developed during the OT&E for the DDG 
Modernisation included the use of the Combat System Trainer (CST) van for 
simulating complex AAW engagements, the use of system modelling and analysis 
by the Weapons Systems Research Laboratory (WSRL), and the use of new ILS and 
RM&A data collection and assessment procedures. 

DDG Modernisation was a low operational risk project. The OT&E 
followed the established RAN policy, however, it was the first to actually implement 
an OT-4 period to more fully assess operational effectiveness and operational 
suitability leading to OA. 

2. Australian Frigate Project 

Two US FFG-7 class frigates, known as Australian Frigates (AF), are being 
built in Australia. The frigates, FFG-06 and FFG-07, are being built to the HMAS 
DARWIN FFG-04 (USN FFG-44) configuration baseline except that the AN/SQS- 
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56 sonar is replaced with the Australian Mulloka sonar. Changes to the configuration 
were kept to a minimum commiserate with safety and operational improvements 
approved for incorporation into FFG-04.[Ref. 51:p. ix] All the identified equipment/ 
system differences between HMAS DARWIN and the AFs had undergone T&E 
prior to installation in FFG-05. 

As the AFs have an almost identical configuration to the last FFG-7 class 
purchased from the U.S., the major risk was in the production, not in the operational 
effectiveness and suitability of the finished product, hence OT&E was limited to an 
OT-3 period. The objective of OT&E for the AFs is to ensure that the user 
requirements have been met, and to provide an estimate of operational effectiveness 
and suitability in the RAN operating environment prior to release of the ship to 
operational service. The OT&E comprises an OT-3 period, which assesses the AF 
against the baseline established by HMAS DARWIN, including any approved 
changes. [Ref. 51 :p. 10] The scope of OT-3 includes a Light Off Examination, Test 
Procedures, CSSQT and Post PSA Trials. OT-3 is being conducted on each ship and 
involves: 

• Installation Inspections 

• Harbour Phase System Qualification Trials 

• Sea Phase System Qualification Trials 

• Communications System Operability Trials 

• Integrated Logistic Support Assessments 

• Fleet Inspections 

• Air Certification 

• Light Off Examination 
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As the significant operational difference in the AFs is the Mulloka sonar, an 
Operational Performance Demonstration (OPD) was planned to determine if the AF 
can fulfil it’s ASW roles and functions with the Mulloka sonar. One week of trials 
was planned during the OT-3 period to ensure that the Mulloka sonar system had 
been correctly installed and met its operational performance characteristics. This 
period was found to be insufficient, primarily due to technical problems, and a 
“mini-OPEVAL” is now planned to quantify the system’s performance. [Ref. 61 :p. 
A-2] The acceptance schedule (Figure 16) follows established RAN policy. 




Figure 16: Australian Frigate Acceptance Schedule 



The TEMP, governing all tests, trials, and evaluations up to AINS, was 
produced as an Australian Frigate Addendum to the U.S. FFG-7 Class TEMP. A Post 
Delivery Test and Trials Plan [Ref. 52] was produced to serve as an advance 
planning document and an implementation plan. 
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An Acceptance Board was established, initially for FFG-05 only. The 
PDT&T Plan includes the requirement that the AB is responsible for providing the 
Final Contract Trials agenda and conducting the trials[Ref. 52:p. 3-17], which is not 
in accordance with RAN AB policy. In its report at AINS, the AB considered that the 
application of the AB process was worthwhile and recommended that it should be 
followed for FFG-06, which is contrary to RAN policy of establishing the Board for 
first of class only. The Board also noted that acceptance documentation was 
contradictory and unclear. [Ref. 61 :p. 2] 

The AF is a low operational risk project, except for the addition of the 
Mulloka sonar. The OT&E followed the established RAN policy, however, it failed 
to plan an OT-4 period to more fully assess operational effectiveness and operational 
suitability of the Mulloka sonar. An OPEVAL to fully assess the performance of the 
sonar has now been included. 

3. ANZAC Ship Project 

Australia and New Zealand are cooperating in the building of the ANZAC 
frigate. Essentially a Blohm and Voss MEKO 200 design incorporating a 
combination of European and US equipment, the design is classified as low risk, 
although modifications to the design were necessary to meet Australian 
requirements and to facilitate construction in Australia. [Ref. 58] 

cl ANZAC Ship OT&E 

OT&E for the ANZAC ships is currently planned to include: 

(1) OT-2 during which RANTAU may witness specific contractor 
testing, and elect to conduct initial operational evaluation using the facilities 
provided as part of the Combat System Support Centre, a LBTS established initially 
for the conduct of DT&E, and then to provide in-service system support. 
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(2) OT-3 consisting of a series of trials conducted to provide an initial 
assessment of the operational effectiveness and suitability of the ANZAC ship 
against the user requirement culminating in AINS. OT-3 will test the critical 
operational issues and will involve sea trials in an operational environment and a 
Ship Qualification Trial (SQT). OT-3 will be conducted on each ship. 

(3) OT-4 consisting of a follow-on series of trials conducted to further 
assess the operational characteristics of the ANZAC ships, particularly as an 
integrated unit in ASW, ASUW, AAW and utility roles. This will be conducted on 
the first of class only and includes participation in two major exercises with multi- 
threat environments, and appropriate freeplay.[Ref. 57] Both OT-3 & 4 will also 
involve an RM&A assessment over a two year period. 

Although only in the early stages, the planned OT&Efor the ANZAC ship 
follows the established RAN policy, with the OT-3 period combined with the SQT, 
being conducted prior to AINS. The ship class has a higher operational risk than 
either the modernised DDG or the AF. 

4. COLLINS Class Submarine Project 

In 1987 the Swedish Kockums Type 471 design was selected to meet the 
requirements for the RAN’s new submarine. The six boats will be the first 
submarines to be constructed in Australia and will have a unique equipment fit. 
Work on Submarine 01 began in February 1990. with the launch planned for 
1993. [Ref. 60] Post delivery T&E activities, proposed by CTE, to be conducted by 
the RAN are outlined in Table 12. 
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TABLE 12: COLLINS CLASS POST-DELIVERY TRIALS [Ref. 59] 



ITT Category 
(per Contract) 


RAN T&E 
Phases 


Subject of ITT / T&E 


CAT 6 


DT-3 


First of Class Trials - Submarine 01 




OT-3 


OPEVAL - Submanne 01 




PAT-6 


Submarines 02 to 06 


CAT 7 


OTA 


Submarines 01,02 and 03 (possibly) 



cl CAT 6 Testing 

CAT 6 testing consists of Safety and Operational Work-ups. First of 
Class Trials, Operational Certification Trials, and OPEVAL. 

(1) First of Class Trials (DT-3). These trials will establish the actual 
performance of the COLLINS design and the in-service safe operating limits and 
conditions for the class. 

(2) PAT-6. These trials will be the Operational Certification Trials 
conducted after delivery in support of AINS of the particular submarine. The criteria 
will be based on applicable thresholds in the Agreed Ship Characteristics and the 
results of First of Class Trials with submarine 01. The principal events in PAT 6 
include: 

• Safety and operational work-up 

• Manoeuvreing trials 

• Weapon handling and discharge 

• Combat system operation, including weapon firings 

• Signature measurements 
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(3) OT-3. These trials are planned to evaluate the first of class only, 
after delivery, conducted in order that, in conjunction with the results of PAT&E 
Part 2, a recommendation for AINS may be made. OT-3 includes an ILS and RM&A 
assessment, and an OPEVAL of four to six weeks during which the submarine will 
be tasked to conduct a series of operations representative of the mission profiles 
against simulated threats and targets of interest. These trials may be reduced after 
experience with the first submarine. 

The proposed CAT 6 schedule is detailed in Figure 17. 
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Figure 17: HMAS COLLINS - Proposed CAT 6 Schedule [Ref. 49] 
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Of note is that the OPEVAL is planned to be conducted following the 
completion of DT-3, work up and the Operational Readiness Evaluation (ORE). So 
it will be conducted on a materially proven and operationally worked up submarine. 

b. CAT 7 Testing 

CAT 7 testing, comprising OT-4 exclusively, will consist of: 

(1) Long Term RM&A and ILS Evaluation. This will determine with 
greater statistical confidence, the Reliability, Maintainability and Availability of the 
submarine class, and assess the implementation of ILS items. 

(2) Measures of Effectiveness (MOEs) Testing. Some MOEs will 
require determination by statistically significant numbers of tests. However, to 
minimise resource requirements the tests will be constructed to allow the total test 
time and asset requirements to be minimised. To accommodate other scheduling 
limitations, this phase will probably occur in a number of stages. 

(3) Major Exercise. This phase assesses the performance of the 
submarine in the roles required of it during a major exercise. 

(4) Operational Deployment. This phase evaluates the performance of 
the submarine and the support infrastructure during an operational deployment and 
maintenance period in a foreign port. 

The proposed CAT 7 schedule is detailed in Figure 18. 
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Figure 18: HMAS COLLINS - Proposed CAT 7 Schedule [Ref. 49] 



c. U se of Land Based T est Sites 

The submarine project includes the construction of a combat system 
simulator and propulsion system simulator, both for system development and 
training applications. Some OT&E activities may be conducted by CTE on these 
systems at the land based test sties on a non-interfering basis to the contractor. [Ref. 
59: p. 20] 

d. Collins Class Submarine OT&E Summary 

Funding provision for RAN T&E up until AINS has not been included 
in the overall financial budgeting for the New Submarine Project [Ref. 59:p. ii]. To 
assist in the progressive development of the TEMP and to plan and agree those 
activities that will be required for AINS and OA, a TEPG has been established. The 
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detailed planning, preparation, execution and reporting of RAN T&E activities is the 
responsibility of a subordinate T&E Working Group. 

The Collins class submarine possibly has the highest operational risk of 
all current major RAN combatant projects. Consequently the increased need for 
OT&E has been recognised by CTE. The planned OT&E program demonstrates a 
departure from the usual RAN OT&E policy, and reflects a growing awareness 
within CTE of the need to separate DT and OT activities, and to include a dedicated 
OT-3 period after the usual SQT and ORE. 

5. OT&E Implementation Summary 

These examples of RAN implementation of ship OT&E policy illustrates a 
number of key issues: 

• The operational risk of ship projects within the RAN is increasing as the new 
RAN combatants have a unique weapons systems configuration. 

• OT&E is not generally conducted to support the production or purchase 
decision of ships. 

• The RAN relies on the results of a combined SQT and OT-3 period to provide 
a basis for AINS. 

• ILS and RM&A assessments are becoming more effective and credible. 

• Modeling and simulation is increasingly being used to assist OT&E. 

• Funding of ship OT&E is not being included in original project estimates. 

• Acceptance and T&E documentation is contradictory and unclear. 

I. RAN OT&E ISSUES 
1. What is OT&E? 

Although the term “OT&E” in the RAN is defined, and the hierarchy of 
Defence and RAN documentation addresses OT&E, the policy for the conduct of 
OT&E in the RAN is not specific and is included with general T&E policy. 
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Combined with a lack of distinction between DT-3, PAT-6 and OTA, this leads to the 
few specific OT&E policy statements often being contradictory and unclear. 

a . Independence of the OT&E Authority 

OT&E is specified in RAN T&E policy [Ref. 44:p. 3] as being 
conducted by an authority independent of the developing and production agencies, 
and the results of OT-3 support an AINS decision. However, CPO Manual [Ref. 
38:p. 14-7] states that the Design Approval Authority (part of the developing 
agency) offers support to trials establishments during OT&E in order to provide 
advice to the Project Director on the suitability of the product for Acceptance into 
Service. This leads to conflicts between the DAA and the Trials Authority during 
the conduct of OT&E. 

b. Confusion between DT-3 , PAT -6 and OT-3 

RAN T&E policy [Ref. 44:pp. 2-3] explains the T&E phases in the 
procurement process and then includes a note stating that OTA and OT-4 will 
incorporate the requirements of PAT-6 and PAT-7 Inspection Tests and Trials as 
described in ABR 1921 (Reference 41). ABR 1921 however, defines many tests as 
being PAT&E, that RANTAU [Ref. 43] define as OT&E. Also DT-3 is shown as 
being conducted during the same period, however, the philosophy, aims, objectives 
and methodology of DT&E, PAT&E and OT&E are totally different. Perhaps this 
overlap or “blurring” of definitions arose because RANTAU was not only the RAN 
OT&E authority, but also the RAN T&E authority who conducted aspects of DT&E 
and PAT&E in addition to OT&E. With a large part of its workload devoted to 
conducting trials following the routine refit of combatants, RANTAU conducted 
HATS and SATS which were understood to be PAT&E and then went on to conduct 
the more operational Ship Qualification Trial (SQT) which was perceived to be 
OT&E. The SQT period, however, is rarely a free play environment, makes limited 
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use of countermeasures, and is conducted prior to the crew being fully worked up. 
Also contractors are often still onboard making final adjustments to systems and 
assisting the crew with operation and/or maintenance. The SQT is essentially a 
PAT&E function which may include some components of the operational 
effectiveness and suitability aspects of OT&E. 

The problem with this “blurring” of definitions goes beyond the 
semantics. The results of OT-3 are used to support an AINS decision, but as it is 
really PAT&E, it assesses more the ability of the ship to meet technical or Fleet 
standards. It does not truly assess the operational effectiveness and operational 
suitability of a new ship against the user requirement. Therefore, a ship is not truly 
subject to OT&E against the user requirement before it is granted AINS. 

This lack of distinction between T&E types also confuses Project 
Directors and other Defence managers. It leads to a lack of consideration of the 
importance of OT&E, and hence the need to consider OT&E funding early in a 
Project. It also leads to confusion over T&E management responsibilities. It 
confuses the role of the Acceptance Board, which is really assuming a quality 
control function over what is essentially PAT&E testing, rather than it’s members 
using their knowledge and experience in assessing the results of true OT&E against 
the user requirement. 

The difference in terminology used by different documents and 
authorities also creates confusion. For example, the term OPEVAL is defined [Ref. 
38:p. 14A-2], but terms such as mini-OPEVAL [Ref. 61], Operational Performance 
Demonstration [Ref. 51] and (Ship) Operational Trials [Ref. 41] [Ref. 44] are not. 

2. OT&E and Acceptance of New Combatants 

With the construction of new frigates and submarines, the RAN has 
embarked on its greatest shipbuilding program since the Second World War. 
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Although these combatants are being built to proven designs and, therefore, are 
considered relatively low risk, they do present a greater operational risk than recent 
RAN combatant programs. In September 1991, CNS expressed concern that the 
RAN “may not be adequately placed” to properly manage the comprehensive and 
complex procedures necessary to accept the ANZAC ships and Collins class 
submarines into service [Ref. 62]. This has focussed attention on the acceptance 
procedures and consequently on the OT&E of the new combatants. 

3. The New RAN OT&E Organisation 

RANTAU ceased to exist as an organisation on May 15, 1992. The rationale 
leading to this decision is important as it is indicative of the level of understanding 
of OT&E within the RAN. 

(L Background 

RANTAU was established in 1966, and originally administered by 
Navy, but later by the Defence Scientific and Technical organisation. In 1982 
control of RANTAU was returned to Navy with the stated purpose of conducting 
operational testing and evaluation, conducting inspections and providing 
operational assessment data.[Ref. 46:p. 1] Functional and operational responsibility 
for RANTAU was transferred from the DCNS Division of Navy Office to the 
Maritime Commander on 31 July 1991 concurrent with the introduction of the 
Program Management and Budgeting System(PMBS). The Director of RANTAU 
was made responsible to the Maritime Commander for the management of 
RANTAU. 

b. What Did RANTAU Achieve? 

As a result of this change in the organisational structure (a change due 
fundamentally to financial funding and accounting requirements) there was a need 
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to question what RANTAU achieved overall, and whether its tasking should be 
amended to more accurately reflect the needs of the Maritime Commander in 
maintaining, and where necessary improving, the operational capability of the 
Maritime Force. A study was directed [Ref. 46] to: 

• focus on the tasks required to be undertaken to assess the operational 
effectiveness of Maritime Force units and the Force as a whole. 

• study any aspects of RANTAU’s current employment which were 
inappropriate given its functional and operational responsibilities to the 
Maritime Command. 

The results of this study [Ref. 47] found that only about 20% of the 
work carried out by the RANTAU Trials Unit was carried out for the Maritime 
Commander. This consisted of SQTs, and HATS and SATS. The large majority 
(80%) of the work done by the Trials Unit was found to be done on behalf of the 
Assistant Chief of Naval Staff - Materiel (ACMAT-N) in accordance with ABR 
1921 (Reference 41). The report stated that the work conducted for ACMAT-N 
included: 

• assisting with the writing of the TEMP for every project. 

• installation inspections as part of PAT&E Part 1. 

• heavy involvement with PAT&E Part 2. 

• the conduct of OT&E for most projects. 

• support for the post delivery acceptance into Naval Service (AINS) 
process. [Ref. 47:p. 17]. 

c. Organisation Options 

The report proposed transferring the Trials Unit, less a proportion to 
support Fleet Trials, to the Materiel Division. The unit would then be ideally situated 
to form the nucleus of an organisation with responsibility for the coordination and 
administration of all trials conducted for and/or by the RAN. It could also form the 
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core of the AB, with perhaps the Director being the standing vice president of the 
body. 

The Materiel Division rebutted this argument with the perception that 
an independent audit authority is necessary to maintain production standards. It 
appears that there is a belief that if the RAM's trials authority was to be a functional 
unit of the Materiel Division, then the head of the Division would be able to 
influence trials results. The report contends that this argument appears to point to a 
management problem rather than a structural issue, but in any case it did nothing to 
counter the argument that the RANTAU Trials Unit is not an appropriate component 
of the Operations Sub-program of the PMBS (i.e., the Maritime Command). A 
solution to this problem would be to return the Trials Unit to the Executive Sub- 
program (i.e.. Navy Office). However, it recognised that the service provided by the 
trials authority was also largely unrelated to that sub-program. The report then 
considered the impact of HQADF, who occasionally tasks RANTAU in support of 
projects for the development of the maritime component of the ADF. 

The report recognised that this seemingly simple organisation of 
RANTAU was, in fact, complex due to the variety of tasks imposed upon the unit 
by a number of masters on a day to day basis. It concluded that as a single unit, 
RANTAU does not rest comfortably with the current arrangement and nor does it 
with any other Navy Sub-program for budgeting and management purposes. 

(L An “Audit” Function 

The report states that RANTAU served an apparent independent audit 
function for the Materiel Command, and that: 

this function was something that had evolved over time rather than in response 
to any specific need. It was beyond the scope of the original purpose for the 
organisation which became RANTAU, and the absolute need for such an audit 
service in the era of PMBS and tight fiscal control is questioned. An audit 
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service for the Materiel Division does not seem an appropriate sub-component 
for the Combat Forces sub-program. 

The report concluded that, other than for a small trials unit to conduct 
trials in support of units assigned to the fleet, it is inappropriate for the Maritime 
Commander to have functional and administrative control of a Trials Unit which 
largely serves the Materiel Division. The options appeared to be the transfer of the 
Trials Unit to the either the DCNS or Materiel Divisions. 

e. The OT&E Function 

The report, however, failed to recognise that RANTAU was the OT&E 
authority. As such, the “audit” work apparently done for the Materiel Command was 
really done on behalf of the sponsor (i.e., HQADF) and the user (i.e., the Maritime 
Command), as OT&E is the final feedback that the item produced by the Materiel 
Command meets the user requirement. 

f Current Trials Organisation 

Following the release of this report, and due to the RAM's decision not 
to adequately staff and fund RANTAU as an independent unit, the functional tasks 
of RANTAU were incorporated within the framework of the Maritime Headquarters 
[Ref. 48]. The current placement of the former Trials Unit, now known as 
Commander Test and Evaluation, within the Maritime Command is shown in Figure 
19. Of note is that the OT&E authority is now responsible to the Maritime 
Commander through the in-service engineering management chain of command. 
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Figure 19: Commander Test and Evaluation Organisation 



J. SUMMARY 

1. Importance of OT&E 

The importance of T&E in general is recognised in the acquisition process, 
to identify problem areas, to allow timely corrective action to be taken, and to reduce 
technical risk. As Australia has a policy of procuring low risk, operationally proven 
systems, which may be constructed locally, rather than higher risk local 
development, the RAN concentrates on post delivery testing, rather than testing to 
support a production or purchase decision. The importance of OT&E to the 
acquisition process, however, is not adequately appreciated. The RAN appears to 
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have its focus on PAT&E to meet technical and fleet standards, rather than an 
assessing operational effectiveness and operational suitability against the user 
requirement. 

2. OT&E Policy 

The lack of emphasis given to OT&E within the RAN is possibly due to the 
lack of clear guidance on OT&E in the T&E policy documentation. The decision to 
subject an acquisition to T&E will initially be decided by the Sponsor and detailed 
in the Capability Proposal, however, the Sponsor and other acquisition managers 
require clear guidance on T&E to enable them to make effective decisions. The 
categories of ship T&E used in the Australian DoD have been derived from the U.S. 
DoD instructions, but interpreted to accord with Australian Requirements. During 
this interpretation there has been a “blurring” of the types of T&E, resulting in a lack 
of clear delineation between the T&E types. This problem was further compounded 
by the adoption of the PAT&E based, USN Total Ship Test Program system, and 
modifying it to include DT&E and OT&E components. 

3. RAN OT&E Organisation 

The RAN and DoD have reorganised over the past five years to create a 
more effective and efficient user requirements organisation, and to better accord 
with Program Management and Budgeting principles. However, as a result, the 
RAN has no T&E, or OT&E policy maker and overseer, no OT&E coordinator 
within Navy Office, and the RAN’s OT&E authority is buried within the in-service 
engineering management area of the Maritime Command. 

4. Understanding of OT&E 

As a result of incomplete, contradictory and confusing T&E policy 
documentation, emphasis on PAT&E and the “blurring” of T&E types, the lack of 
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true OT&E actually conducted and few personnel experienced in OT&E. the 
understanding of OT&E within the RAN is poor. As a consequence, the philosophy, 
methodology, and benefits to the RAN of OT&E are not adequately recognised. 

5. Summary 

The demise of the RAN's independent OT&E authority, RANTAU, and its 
subsequent integration within Maritime Headquarters as the Commander, Test and 
Evaluation (CTE), was the latest in a series of reorganisations that has significantly 
reduced the capability and authority of the OT&E community. The new combatants 
currently under construction for the RAN, although relatively low risk by USN 
standards, are a higher operational risk than previous RAN ship programs. Although 
using proven technologies, both the ANZAC ships and Collins class submarines 
have unique weapon and sensor fits, not currently in service with large parent navies. 
The conduct of PAT&E only, is not considered sufficient to measure the operational 
effectiveness and suitability of these new combatants to determine, with confidence, 
that they meet the user requirement. This can be achieved by the conduct of an 
effective OT&E program. 
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V. USN/ RAN OT&E COMPARISON 



This chapter compares the characteristics of the USN and RAN OT&E systems 
and highlights significant differences. 

A. DEFENCE ACQUISITION PROCESS 

The Defence system acquisition life cycle of both nations consists of 
progressive development phases separated by major decision milestones. 
Milestones are periodic formal program reviews for authorisation to advance to the 
next stage. The two systems are compared in Figure 20. With a policy to maintain 
world leadership in key Defence technologies, the U.S. places priority on 
development programs. The major decision point is centred on the decision to 
proceed beyond Milestone IIIA - Low Rate Initial Production , when the large 
dollars in an acquisition program are due to be spent in the production phase. 
Passing a formal, independent OPEVAL is its major criterion for approval to Full 
Rate Production. 

Although the decision to produce or purchase a new system is important, the 
Australian DoD prefers lower risk, proven technologies for the majority of new 
Defence acquisitions. In many cases, systems with the fundamental capabilities 
required by the ADF are available overseas, however, some of these systems require 
additional development, adaptation or integration to meet Australian specific 
requirements. The Australian system, therefore, places a high priority on the post 
production phase with the three milestones of Preliminary Acceptance (PA), 
Acceptance into Naval Service (AINS) and Operational Acceptance (OA). 
However, Australia is now buying far less “off the shelf” from overseas, and 
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systems are increasingly being designed and built in Australia, or at least adapted 
from overseas designs. 
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Figure 20: Comparison of U.S. and Australian Acquisition Processes 



B. T&E DEFINITIONS 

Definitions and terminology form the basis of a T&E system. As the categories 
of ship T&E used in the Australian DoD have been derived from the U.S. DoD 
instructions, both Navies recognise three types of T&E: 
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• Developmental Test and Evaluation (DT&E) 

• Production Acceptance Test and Evaluation (PAT&E) 

• Operational Test and Evaluation (OT&E) 

The categories within these types of T&E, however, have been interpreted by 
the RAN to accord with Australian requirements [Ref. 38:p. 14-1]. This 
interpretation leads to some subtle, but significant, differences in T&E definitions 
between the two Navies. 

1. Developmental Test and Evaluation (DT&E) 

DT&E is conducted to assist the engineering design and development 
process and to verify attainment of technical performance specifications and 
objectives [Ref. 1 l:p. 3]. Both Navies agree that DT&E is conducted in three major 
phases. Table 13 compares these DT&E phases. Minor wording differences aside, 
the three DT&E phases are essentially identical. The most significant difference is 
that the USN recognises PAT&E as a form of DT-3 testing. 



TABLE 13: COMPARISON OF DT&E PHASES 



USN Description 


DT 


RAN Description 


Demonstration that all technical risk 
areas have been identified & that best 
technical approaches have been 
accepted. 


DT-1 


Validation of design concept. 


Demonstrate that design meets 
specifications. 


DT-2 


Proving design. 


Testing conducted on production items 
to ensure compliance with contracted 
specifications. PAT&E is a form of 
DT-3. 


DT-3 


Demonstration that production meets 
required technical characteristics. 
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2. Production Acceptance Test and Evaluation (PAT&E) 



Both Navies agree that PAT&E is testing conducted on production items to 
ensure systems meet contract specifications and requirements. PAT&E for ships in 
the USN is managed under the Total Ship Test Program (TSTP) which includes 
seven phases of PAT&E. In recent years the RAN has adopted the TSTP, initially 
with the modernisation of the DDGs, and now in the testing of the locally produced 
FFG-7 class under the Australian Frigate Project. The adoption of the TSTP by the 
RAN included the redefinition of the test phases. A comparison of the PAT&E 
phases is provided in Table 14. There are major differences between the two Navies 
in these PAT&E categories. The USN system includes only PAT&E, whereas the 
RAN includes elements of DT&E and OT&E, in addition to PAT&E, and they are 
given PAT designations. 



TABLE 14: COMPARISON OF PAT&E PHASES 



USN Description 


PAT&E 


RAN Description 


Not defined in U.S. system 


PAT-0 


Design and engineering development 
tests 


Material receipt inspection and shop 
tests 


PAT-1 


Production and bum-in tests 


Shipboard installation inspections 
and tests 


PAT-2 


Environmental qualification tests 


Equipment level operational tests 


PAT-3 


System development tests 


Intrasystem tests 


PAT-4 


Harbour testing 


Intersystem tests 


PAT-5 


Sea testing 


Special tests e.g., surveys, rangings 


PAT-6 


Operational and Qualification Trials 
(conducted with OT-3) 


Trials tests including sea trials, 
builders trials, acceptance trials. 


PAT-7 


Follow on Operational T&E 
(conducted with OT-4) 
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3. Operational Test and Evaluation (OT&E) 

Both Navies concur that OT&E is conducted to determine a system’s 
operational effectiveness and operational suitability, to identify system deficiencies 
and the need for potential modifications to meet established OT thresholds, and 
develop tactics. They also agree that OT&E has three distinguishing characteristics: 

• It is conducted in an operationally representative environment 

• It is conducted on production representative equipment using fleet personnel 
for operation and maintenance. 

• It is conducted against a threat-representative simulated enemy carrying out 
threat tactics per the latest threat assessment. 

They both use the term “Operational Evaluation”, abbreviated as 
“OPEVAL” to cover T&E on production representative baseline equipment using 
the maintenance and support personnel and equipment for normal operational use 
which aims to: 

• Demonstrate operational effectiveness and suitability. 

• Provide data to assist in the development of tactical aspects of the equipment. 

• Verify data, handbooks and documentation covering the operation of the 
system. 

The USN uses the term OPEVAL as the final stage of OT-II, supporting a 
production decision. Although the Australian definition of OT-2 includes an 
OPEVAL, the term is not used exclusively for OT-2. It is used in many contexts 
from OT-2 to OT-4. 

There are also significant differences in the detailed OT&E definitions. The 
RAN includes ship Certification / Qualification trials in OT-3, whereas the USN 
categorises Certification trials as PAT&E and Ship Qualification Trials (SQT) as 
regular in-service PAT&E conducted by the INSURV Board and Type Commander, 
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not as OT&E. A comparison of the OT&E phases of the USN and RAN are given in 
Table 15. 



TABLE 15: COMPARISON OF OT&E PHASES 



USN Description 


OT 


RAN Description 


Early Operational Assessment 


OT-I 


Operational assessment of the 
development proposal 


OT&E conducted to support a 
production decision. Final phase is 
termed OPEVAL. 


OT-2 


Demonstration of achievement of 
program requirements for operational 
effectiveness and suitability of a 
prototype (OPEVAL) to support 
proceeding to full production 


OT&E conducted on production 
system to verify correction of 
deficiencies after OPEVAL, and 
certification of operational 

effectiveness and suitability. 


OT-3 


Demonstration of achievement of 
program requirements for operational 
effectiveness and suitability on 
production of ship/aircraft/system, 
normally in independent operations 
using normal Fleet personnel. 
Certification/Qualification Trials 
include limited reliability, 

maintainability, availability and 
logistic supportability assessments. 


Validation of the operational 
effectiveness and suitability of 
production systems. Usually in 
different environments and to assess 
integrated operation of system. For 
ship programs, is normally conducted 
to verify that critical deficiencies have 
been identified and to complete any 
outstanding OT&E. 


OT-4 


Demonstration of achievement of 
program requirements for operational 
effectiveness and operational 
suitability on production of ship/ 
aircraft/system using normal Fleet 
personnel in a multi-force, multi-threat 
environment. Includes detailed 
reliability, maintainability, availability 
and logistic supportability 
assessments. 


Not defined in the U.S. System. 


OT-5 


Follow-on OT for assessment after 
modernisations, new applications or 
defect rectifications after OTA 
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C. T&E IN THE ACQUISITION PROCESS 

Both Navies appreciate the importance of T&E in the acquisition process. T&E 
results are assessed as part of the milestone reviews. However, the level of 
importance of T&E, and OT&E in particular, differs greatly. 

1. The Requirement for T&E 

The importance of T&E, and OT&E in particular, to the progressive 
assessment of programs in the USN is such that its use has been mandated by the 
U.S. Congress and incorporated into the laws of the United States. The law includes 
the provision of a Director of Operational Test and Evaluation (DOT&E) in the 
Department of Defence. The law also addresses specific areas of OT&E reporting 
and conduct to ensure the Congress is kept informed, and the testing and reporting 
are impartial [Ref. 17]. By including the requirement that a major Defence 
acquisition program may not proceed beyond low-rate initial production until initial 
OT&E of the program is completed. Congress ensures that OT&E is an integral part 
of the acquisition process. 

The USN appreciates that, if adequate OT&E is not done, and the weapon 
system does not perform satisfactorily in the field, significant changes may be 
required. Moreover, the changes will not be limited to a few developmental models, 
but may also be applied to items already produced and deployed. The USN also 
recognises that, in extreme situations, it also risks deploying systems which cannot 
adequately perform significant portions of their missions, thus degrading its 
deterrent / defensive capabilities and endangering the safety of military personnel 
who operate and maintain the systems. 

The need for local T&E is recognised by the Australian Government and 
DoD, for both indigenous and overseas systems, however, the importance of OT&E 
in particular, is not specifically addressed in any Australian T&E policy document. 
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Perhaps this lack of recognition of the importance of OT&E is due to the RAN 
procuring low risk, complete ships in the past (e.g., FFG-7 class) with the wealth of 
USN DT&E and OT&E behind it. Also the RAN, unlike the USN, has not 
experienced the operational situations in recent years which may have highlighted 
operational effectiveness and suitability deficiencies in its ships. With the RAN now 
buying higher risk, unproven combinations of systems within its ships, it must now 
rely on its own OT&E. 

2. T&E Contributions At Major Milestones 

The USN OT&E system forms an integral part of the acquisition process. 
Program advance from one phase to the next is not by the calendar of planned 
schedule, but by actual resolution of critical operational issues and achievement of 
pre-set thresholds verified by T&E [Ref. 1 l:p. 2]. OT&E in the RAN, on the other 
hand, may be conducted during each phase in the acquisition process, but it is not 
essential to a local system meeting a development or production milestone, or the 
local production or purchase decision of an overseas system. It is however, required 
before a system achieves Acceptance into Naval Service or Operational Acceptance. 
Figure 21 compares the contributions of T&E to the acquisition milestones for each 
Navy. 

Although the diagrams appear to be very similar, the RAN OT&E 
contributions to the acquisition milestones are by no means definitive. There is no 
documented requirement for the conduct of OT-1 and /or OT-2 prior to a production 
or purchase decision of major acquisitions. The decision to conduct OT&E being the 
responsibility of the Sponsor, Project Manager and the Defence acquisition 
committees. RAN T&E policy documentation almost exclusively focuses on the 
post production acceptance phases. Even during the acceptance phase, however, the 
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requirement for OT-3 is clouded in definition differences between policy documents 
The RAN appears to have its focus on PAT&E. 
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Of note is that the USN diagram shows separate activities for DT&E and 
OT&E related tests, while the RAN diagram groups them all together. This 
demonstrates the delineation between DT&E /PAT&E and OT&E evident in the 
USN system. 

D. OT&E ORGANISATION 

The OT&E organisation consists of a number of functions. The authorities 
performing these functions in each Navy are listed in Table 16. 



TABLE 16 : COMPARISON OF OT&E AUTHORITIES 



USN 

Authority 


OT&E 

Function 


RAN 

Authority 


Director, 


Policy Maker 




Operational Test & Evaluation 
(DoD) 


& Overseer 


Nil 


N8 - Warfare Requirements 


Sponsor 


Director General Force 


Chief of Naval Operations 




Development (Sea) 


Commander, 


Developing 


Assistant Chief of Naval Staff - 


Naval Sea Systems Command 


Agency 


Materiel 


Commander, Operational 


OT&E Tester 


Commander, Test & Evaluation 


Test and Evaluation Force 


& Evaluator 




Ships Commanding Officers, 


User 


Ships Commanding Officers, 


Fleet Commanders 




Maritime Commander 


Director, 


OT&E 




Test, Evaluation & 
Technology Requirements 


Coordinator 


Nil 
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1. OT&E Policy Maker and Overseer 

An OT&E policy maker and overseer is employed in the U.S. system 
responsible for policy formulation, evaluation and oversight of all OT&E. The 
Director of OT& E (DOT&E) is the principal OT&E official in the Department of 
Defence and the principal advisor to the Secretary of Defence on OT&E. In addition, 
a Director T&E, responsible for developmental testing policy reports, to the 
USD(A) in DoD. 

There is no clear OT&E, or T&E, policy maker or overseer within the 
Australian DoD or the RAN. The Chief of Capital Procurement [Ref. 38] addresses 
T&E in general, but more in the form of guidance rather than policy. The former 
Director General of Naval Warfare (DGNW) drafted the RAN Test and Evaluation 
Policy Defence Instruction [Ref. 44], however after the recent reorganisation, T&E 
policy for the RAN is presently in a state of flux, with no office having assumed the 
responsibility [Ref. 50]. 

2. Program Sponsor 

The program sponsor is responsible for the development of concepts for 
operations for naval warfare systems, and for acquisition program requirements and 
related system thresholds. He/She is also responsible for the subsequent trials and 
continuing overview to ensure that the equipment meets the requirement. Within the 
USN, major warship programs are sponsored by N8 - Warfare Requirements, under 
CNO. Under the new USN organisation, increasing importance is given to the Fleet 
and Type Commanders in determining requirements and allocating priorities. 
Besides generating the user requirement, the sponsor also has a review function in 
the OT&E process. [Ref. 20] 

The Australian Defence Force has a centralised force development and user 
requirements organisation, within HQADF, the Director General Force 
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Development (Sea) having sponsorship of major RAN acquisitions. The decision to 
subject an acquisition to T&E in the RAN is initially decided by the sponsor and 
detailed in the capability proposal [Ref. 44:p. 1]. 

3. Developing Agency 

When a program is approved, a program office is formed to manage the 
acquisition of the system. In the case of the USN, this office is established within the 
Naval Sea Systems Command (NAVSEA). The program office is responsible for the 
conduct of DT&E in preparation for OT&E. To assist all NAVSEA programs with 
T&E matters, NAVSEA have established their own T&E office (SEA 60). Although 
primarily a DT&E policy authority, this office also has an input into the OT&E 
process by reviewing the TEMP, and by conducting OT&E readiness reviews of 
NAVSEA programs. 

Within the RAN, the Materiel Division is responsible for acquisition of the 
capital equipment to meet the requirement. Similar to the USN, the nature and extent 
of the T&E to be conducted is decided by the Project Director in consultation with 
the Design Approval Authority and RANTAU. However, the Materiel Division does 
not have a T&E office to support Project Directors. 

4. OT&E Tester and Evaluator 

The Operational Test and Evaluation Force (OPTEVFOR) is the USN’s 
sole independent test agency responsible for the planning and conduct of OT&E. 
COMOPTEVFOR reports directly to the CNO, and is separate and distinct from the 
developing and procuring commands. 

The RAN Trials and Assessing Unit (RANTAU) was the RAN’s OT&E 
authority who also conducted DT&E / PAT&E on behalf of Project Directors and 
Ship Qualification Trials for the Maritime Commander. With the disestablishment of 
RANTAU in May 1992, these trials functions were transferred to the Commander, 
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Test and Evaluation (CTE) under the Chief Staff Officer (Engineering) in the 
Maritime Command. 

5. User 

The Commanding Officer and crew of the ship under test, and his Fleet and 
Type Commanders are the ultimate users of the system. The Maritime Commander, 
and the individual ships’ Commanding Officers and crews are the ultimate users of 
a ship. The personal assessment of the Commanding Officer of the ship under test is 
sought by OPTEVFOR during the conduct of OT&E [Ref. 15:p. 44]. The 
Commanding Officer of a ship under test usually provides CTE with his opinions on 
the ship’s performance at the conclusion of a trials period. 

6. Coordinator 

CNO has responsibility for ensuring the adequacy of the Navy’s overall test 
and evaluation program. T&E policy and guidance are exercised through the 
Director, Test, Evaluation and Technology Requirements (N091). This organisation 
also acts as a T&E Focal Point and Coordinator, responsible for coordination of 
T&E matters in the designated Programs, System Commands and Department of the 
Navy. [Ref. 13:p. 7-5]He also chairs the Test and Evaluation Coordination Group 
The RAN currently has no authority with this role, however, the T&E manager 
within major projects is responsible for chairing the Test and Evaluation Planning 
Group. 

7. OT&E Organisation Summary 

A comparison of the OT&E organisation structures of both Navies is shown 
in Figure 22. The USN OT&E organisation is well structured with comprehensive 
responsibilities for each participant, and clearly delineated from the DT&E 
structure. COMOPTEVFOR is the truly independent tester and evaluator, being 
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separate from developing, production and user agencies, and responsible only for 
OT&E. As a Rear Admiral and reporting directly to the CNO, he has authority, 
visibility and influence. The office of DOT&E provides a layer of oversight on the 
OT&E process to ensure objective testing is conducted, giving credible results and 
leading to impartial decisions. NAVSEA has its own T&E agency to assist PM’s in 
DT&E, and in preparation for OT&E. The OT&E process is coordinated from 
within CNO’s office by the Director, Test, Evaluation and Technology 
Requirements. 

Following a number of recent reorganisations, the RAN’s OT&E structure 
is in disarray. The RAN’s OT&E authority is now CTE within the Maritime 
Command. In addition to OT&E responsibilities, he also conducts some DT&E, and 
some functions equivalent to the USN INSURV board, OPTEVFOR and the Type 
Commander. Only of Commander rank, he lacks authority, particularly with Project 
Directors and, being within the in-service engineering area, he possibly lacks 
credibility and influence on operational matters pertaining to OT&E. In developing 
programs, the RAN relies on the minimal interface between the centralised sponsor 
(DGFD(Sea)) and CTE to ensure appropriate OT&E is considered, since the sponsor 
has no agency within his own organisation to provide the necessary advice. Once a 
project is established, the Project Director performs the coordinating role through 
the TEPG. The Materiel Command, however, has no T&E agency to advise PD’s on 
this role, and to be the retainers of “lessons learned” from other projects. Finally, 
with no clear policy maker for T&E in general, or OT&E in particular, the RAN 
lacks the ability to effectively manage overall T&E, let alone OT&E, within the 
RAN. 
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USN 




Figure 22: OT&E Organisation Comparison 
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E. OT&E POLICY 

USN OT&E policy has its basis in statutes approved by Congress. The RAN has 
no such formal basis and its OT&E policy is embedded in the more general T&E 
policy. A number of OT&E policies of interest include: 

1. OT&E Documentation 

USN policy is refined in the DoD acquisition Directives 5000. 1 [Ref. 25] 
and 5000.2 [Ref. 24]. The essential T&E elements in these directives are further 
detailed in OPNAV Instruction 3960. 10C [Ref. 11] and other subordinate 
instructions. USN OT&E documentation is comprehensive and provides clear, 
consistent and non-conflicting policies and guidance to OT&E participants. This 
concise documentation establishes the disciplined management approach to OT&E 
taken by the USN. 

The RAN does not enjoy this sound basis of OT&E policy. Some general 
guidance to assist Project Directors in assessing the need for T&E in a project and 
to provide guidelines for its conduct is provided in the Capital Equipment 
Procurement Manual (CEPMAN 1 ) [Ref. 38]. The top level T&E policy document 
for the RAN is the Defence Instruction (Navy) on RAN Test and Evaluation Policy 
[Ref. 44], while ABR 1921[Ref. 41], contains T&E and acceptance policy 
applicable to ships building, undergoing modernisation, conversion or extended 
refit. These documents provide broad T&E policy only and, other than defining 
OT&E, provide little policy as to its conduct. 

2. OT&E Requirements 

In the USN system, the requirement to conduct OT&E on new major 
acquisitions is mandatory. Although the Sponsor initially develops the system 
thresholds, the level and extent of OT&E required is delineated in the TEMP and 
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approved prior to Milestone I. The appropriate level of OT&E is determined by 
COMOPTEVFOR in consultation with the Program Manager, with DOT&E 
oversight. This ensures that OT&E planning is conducted and approved early so the 
costs of OT&E can be included in project funds. 

In the RAN, the decision to subject an acquisition to T&E is initially 
decided by the Sponsor and detailed in the Capability Proposal. The nature and 
extent of the T&E (no distinction is made in RAN policy between the various types 
of T&E) is decided by the Project Director in consultation with the Design Approval 
Authority and CTE. Any conflict over the requirement, nature and/or extent of the 
proposed T&E is resolved by DCNS and ACMAT-N for operational and design 
matters respectively. [Ref. 44:p. 2] The role of DCNS should now rest with 
DGFD(Sea) as the Sponsor. The determination of the OT&E required has 
traditionally not been achieved early enough in the life of a project, and so project 
funding has not included the provision for OT&E. 

3. Independence of OT&E Authority 

As OT&E is the final exam for an acquisition program, independence is 
necessary so that there is no question of impropriety. The USN requires its 
independent operational test activity to be separate and independent from the 
materiel-developing and procuring agency and the using agency. This ensures that 
the testing and reporting agency has nothing to gain or lose by whatever they say. As 
major acquisition decisions hinge on the results of OT&E within the USN, 
independence is necessary to ensure that the reports on which those decisions are 
based are not self serving. [Ref. 19] 

Although major acquisition decisions within the RAN are less based on 
OT&E, the RAN also maintains the policy that OT&E is to be conducted by an 
authority independent of the development and production agencies. With the demise 
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of RANTAU and the transfer of OT&E responsibilities to CTE on the staff of the 
Maritime Commander, independence from the development and production 
agencies is achieved, but not from the end user as in the USN. The implications of 
this lack of independence are that the end user may have his own aims and objectives 
which may conflict with the total impartiality of OT&E reporting. The politics of 
requirements and funding could come into play, as the Maritime Commander 
establishes scheduling priorities for his resources, and OT&E may take a low 
priority behind operations and training. The Maritime Commander may also not 
wish to “stress” the system, in case he perceives that operational deficiencies so 
highlighted may be seen as having some reflection on his own organisation. 

4. Contractor Involvement in OT&E 

To ensure impartiality and credibility of testing, the USN prohibits persons 
employed by the contractor for the system being tested being involved in OT&E, 
unless contractors are planned to be involved when the system is deployed in 
combat. 

The RAN requires that during OT&E, a system should be demonstrated 
“using normal Fleet Personnel”, however, the policy documentation makes no 
statement regarding contractor personnel involvement in system operation or 
maintenance during OT&E. It has been RAN practice (e.g., in the DDG 
Modernisation Project) to have contractors onboard resolving technical problems 
during the SQT period, which is included as part of OT-3 in the RAN, thereby 
possibly causing test bias. 

5. T&E Planning 

Both the USN and RAN use the Test and Evaluation Master Plan (TEMP) 
as the single, executive, long range planning document for T&E in a project. Both 
Navies are in agreement that approval of the TEMP constitutes establishment of a 
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contract between the requirement-setter and the PM which is essential to formal and 
accountable program execution, and between the requirement-setter and the OT 
agency, upon which the OT agency will independently evaluate system operational 
effectiveness and suitability. The main differences between the two Navies are in the 
responsibility and level of approval, and in the timing of the TEMP. 

cl TEMP Responsibility 

Within the USN, the TEMP is drafted by the PM. COMOPTEVFOR 
drafts Part Four on OT&E and provides OT&E resource requirements for Part Five. 
The TEMP is then co-submitted by both the Program Manager and 
COMOPTEVFOR. It is reviewed by the Assistant Chief of Naval Operations 
(Surface Warfare), the Director, T&E and Technology Requirements, and the 
Assistant Secretary of the Navy (Research, Development and Acquisition). It is 
ultimately approved at the DoD level by the Director, OT&E and the Director, Test 
and Evaluation. 

Within the RAN, the TEMP is the ultimate responsibility of the project 
sponsor [Ref. 44:p. 4]. However, during the period when there is a full time Project 
Office, TEMP coordination is normally carried out by the T&E Manager in the 
Project Office. CTE as the OT&E authority provides the OT&E input. The TEMP 
is endorsed by all affected organisations and is released jointly by the Deputy Chief 
of Naval Staff (DCNS), and the Assistant Chief of Naval Staff-Materiel (ACMAT- 
N), as the Developing Agency. If any issues remain unresolved at this level, the 
TEMP will be resolved by the Chief of Naval Staff (CNS). [Ref. 44:pp. 4-5] The 
authority for DCNS to release the TEMP was based on his role as Project Sponsor. 
With HQADF now assuming this role, the DCNS function in TEMP development 
is unclear. 
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In the USN, drafting the TEMP is the responsibility of the PM while the 
DoD heads of OT&E and DT&E authorities approve the TEMP. In contrast, in the 
RAN, the Sponsor has ultimate responsibility, although the TEMP is managed by 
the PM, and the TEMP is approved by the heads of the developing agency and the 
(former) Sponsor. 

b. TEMP Timing 

The TEMP is required in the at Milestone I- Concept Demonstration 
Approval within the USN, and is updated for each subsequent milestone and when 
significant program changes occur. The RAN requires the TEMP to be raised in 
draft form as early as practicable, but in actuality the TEMP for major projects is 
usually drafted after production approval (Milestone 3). 

6. Combined DT&E and OT&E 

“Combined testing” refers to a single test program conducted to support 
both DT&E and OT&E objectives. The advantages of combined testing is the 
shorter time required for testing, and cost savings by eliminating redundant 
activities. These need to be weighed against the limitations of the additional 
extensive coordination required and the less than optimum environment and 
coverage for OT&E that may occur. Early involvement of OT&E personnel during 
system development increases their familiarity with the system and permits 
identification of operational concerns early in the program. 

The USN has a policy that combined DT&E and OT&E testing should be 
considered when there are time and cost savings [Ref. 24:p. 8-3]. However, this 
combined approach must not compromise either the developmental or operational 
test objectives. Also a final independent phase of operational testing and evaluation 
is required for beyond low rate initial production decisions [Ref. 24:p. 8-4]. The US 
has extensive experience with combined testing, some successful and some 
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unsuccessful, and they conclude that it is possible to have combined test teams 
involved throughout the testing process. The DT&E and OT&E teams can share 
mutually beneficial data, as long as the test program is carefully planned. However, 
evaluation and reporting activities must be conducted separately. [Ref. 18:pp. 17-1 
- 17-4]. 

The conduct of combined DT&E, (PAT&E) and OT&E is not addressed in 
RAN T&E policy, however, it is implied as being almost a requirement. The T&E 
DI(N) [Ref. 44:p. 3] states that OT-3 and OT-4 will incorporate the requirements of 
PAT-6 and PAT-7 Inspection Tests and Trials as described in ABR 1921. It also states 
that OT-3 will be conducted during Ship Qualification T rials. The RANs "blurring" 
of the distinction between the types of T&E leads to DT&E and OT&E often being 
conducted over the same period, by the same test team. Although the RAN 
recognises, by definition, the difference in objectives and methodology between the 
types of T&E, no limitations or guidance as to the possible hazards of this combined 
testing approach are addressed. 

7. Modeling and Simulation 

The use of modeling and simulation can increase the efficiency of the T&E 
process, reduce the time and cost, provide otherwise unattainable and 
unmeasureable data, and provide more timely and valid results. The USN 
encourages the use of simulation and modeling to assist in projecting operational 
effectiveness and operational suitability prior to Milestone II - Development 
Approval, but limit the use of simulation and modeling in subsequent OT&E to that 
of supplementing OT&E test data [Ref. 15:p. 2-37]. 

The use of models and simulation in OT&E is not addressed by the RAN 
documentation, although modeling and simulation performed by the Maritime 
Systems Division of the Defence Science and Technology Organisation were used 
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by RANTAU to assist the evaluation of the upgraded combat and weapons systems 
in the Modernised DDGs. 

With the increasing costs of weapons systems, decreasing budgets and the 
subsequent need for efficiency in the conduct of OT&E, simulation can be used 
during OT&E to extend test results, to improve the statistical sample, or to 
determine overlooked or directly unmeasured parameters. This is particularly 
relevant to the RAN who lack appropriate ranges and the financial resources for 
extensive test programs of expensive weapons. A policy directing the use of models 
and simulation in OT&E in the RAN is required. 

8. Foreign Weapons Evaluation 

The US DoD has a foreign weapons evaluation program which is designed 
to support the evaluation of a foreign nation’s weapons system, equipment or 
technology in terms of its potential to meet a specific U.S. military requirement. The 
primary objective of the program is to reduce the costs of research and development, 
while leading to the acquisition of foreign equipment for U.S. use [Ref. 18:p. 21-1]. 
From the OT&E viewpoint, the USN ‘try-before-buy policy’ is still maintained, 
despite the early phases of OT being unable to be achieved. When procurement of a 
foreign weapon system is planned, the developing agency and COMOPTEVFOR 
may be directed to assess the adequacy of any previously conducted DT&E and 
OT&E and to provide recommendations on the need for additional T&E prior to 
procurement [Ref. 15:p. 2-34], 

Despite the most recent Australian Government “White Paper” on Defence 
Policy [Ref. 6:p. 70] requiring the need to be able to determine the performance in 
the Australian environment of equipment of both overseas and local origin, no DoD 
or RAN T&E policy specifically addresses the evaluation of foreign systems. Prior 
to a major acquisition, project teams review possible contractors’ proposals and 
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evaluate them against the user requirement. OT&E personnel may be part of these 
teams, however, there is no policy as to the OT&E authority’s role in foreign 
weapons evaluation within the RAN. 

9. OT&E Funding 

OT&E is an expensive process and generally occurs late in the life of a 
project when money is usually tight. Within the USN, funding associated with T&E 
(including instrumentation, targets and simulations) are identified in the system 
acquisition cost estimates, acquisition plans and the TEMP [Ref. 18:p. 18-11]. The 
Program Manager plans, programs, budgets and funds the costs of all resources 
identified in the approved TEMP for all T&E through OT-III. Funds required to 
conduct OT&E are programmed and budgeted by OPTEVFOR and advised to the 
Program Manager. OT&E costs include test articles, expendables, targets, data 
collection and reduction and OPTEVFOR program related costs. The Program 
Manager does not fund fleet operating costs for T&E support, which includes fuel 
and aircraft. These costs for OT-II and III, and all costs for OT-IV, except 
procurement costs and OPTEVFOR costs, are funded by the Fleet CINCs.[Ref. 
1 l:p. 21] The RAN employs a similar funding arrangement where OT&E is funded 
separately within a project and is estimated and bid for by CTE.[Ref. 44:p. 6] 

The essential difference is that the USN develops the TEMP, and hence 
identify the resources required and their costs, earlier in a project’s life than does the 
RAN, thus enabling the project to include these requirements in project funding. 

10. Land Based Test Sites 

A Land Based Test Site (LBTS) is a facility that duplicates, simulates or 
stimulates the employment of a system’s planned operational installation and 
utilisation. Used primarily for the purpose of conducting DT&E, it is sometimes 
used to test system integration and overall performance. The USN often use LBTS 
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for the conduct of initial OT&E to gain an estimation of potential operational 
effectiveness and suitability, primarily to support Milestone IIIA (Low Rate Initial 
Production) decisions and not the more stringent Milestone IIIB (Full Rate 
Production). 

RAN guidance for OT&E using land based test sites includes a statement 
that system centres and simulators will be employed for early stages of OT&E if 
available [Ref. 44:p. 4], and that OT-2 may be conducted at a land-based test site 
[Ref. 44:p. 3]. Although LBTS are being developed for both the ANZAC frigate and 
COLLINS submarine projects, OT-2 in support of production decisions will not be 
conducted. The use of these LBTS for later OT&E was not planned in the original 
project schedules, and so is subject to them being used on a non-interfering basis to 
the contractor. 

The USN has demonstrated that early OT&E on LBTS can give an 
estimation of potential operational effectiveness and suitability, and hence identify 
potential operational problems early and minimising operational risk. Their use for 
OT&E during the ANZAC frigate and COLLINS submarine projects would appear 
to offer similar advantages to the RAN. 

11. Ship OT&E 

Both the USN and RAN recognise that ship acquisition projects will be 
treated differently to others in so far as protracted construction time usually 
precludes T&E of a prototype before the decision to proceed to production. Because 
the development and construction period for a major ship in the USN, normally 
precludes completion of initial OT&E on the lead ship prior to the production 
decision for follow-on ships, successive phases of OT&E are accomplished as soon 
as practicable to reduce risk and minimise the need for modification to follow-on 
units. Ship acquisition programs, therefore, usually have OT&E between Milestones 
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II and III, which consist of individual weapon systems testing and system integration 
at land based test sites. 

With the construction of lower risk, proven overseas ships adapted to 
Australian requirements, the RAN assessed the degree of technical risk as low. 
OT&E between Milestones II and III in the RAN was, therefore, not considered 
necessary. 

F. SHIP ACCEPTANCE 

Both Navies have similar policies and procedures for acceptance of ships from 
contractors and then into service. However, only the USN states the purpose of ship 
acceptance which is “to ensure delivery to the Fleet of complete ships, free from 
both contractor and government responsible deficiencies" [Ref. 35:p. 1]. 

Each Navy uses similar terminology, but these have different meanings which 
can create confusion. A comparison of terms applicable to ship acceptance is given 
in Table 17. 

1. Ship Acceptance Policy 

Ship acceptance and Fleet Introduction within the USN has traditionally 
been based only on the successful completion of PAT&E and material inspections. 
However, with the treatment of ships as complete systems, as for the DDG-51, Fleet 
Introduction is now based also on OT&E results. 

The RAN has also placed emphasis on PAT&E leading to Acceptance into 
Naval Service (AINS), although OT-3 has been included in the trials program 
leading to AINS. However, these trials conducted as OT-3, are really PAT&E / 
DT&E with minimal true OT&E. 
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TABLE 17: COMPARISON OF ACCEPTANCE TERMINOLOGY 



USN Term 


USN Definition 


RAN Term 


RAN Definition 


Acceptance 


The legal act of accepting 
custody of a new construction 
ship by the Navy upon delivery 
by a priv ate shipbuilder. 


Delivery 


When the ship is contractually 
delivered from the 

contractor’s ownership to the 
Government. 


Delivery 


The actual assumption of 
custody by the Navy incident 
to acceptance. The date of 
delivery from a private 
shipyard is also the date of 
acceptance. 


Preliminary 

Acceptance 

(FA) 


Determination that the ship has 
reached both material and 
support status sufficient to 
safely and effectively proceed 
with Navy trials. It is usually 
concurrent with Delivery. 


Fleet 

Introduction 


Approval by SECNAV that a 
ship class meets the 
operational effectiveness, 
operational suitability, safety 
and material standards for 
service use. 


Acceptance 
into Naval 
Service 
(AINS) 


Acceptance of the ship by CNS 
as a fully operational unit of 
the Fleet, based on operational 
performance and support 
levels obtained at that time. 






Operational 

Acceptance 

(OA) 


Carried out only of first of 
class to supplement the AINS 
assessment using additional 
operational and support 
experience with the Fleet to 
refine the vessel and Naval 
Requirements. 



2. Ship Acceptance Procedure 

The procedure for USN ship acceptance and introduction to the Fleet as 
used for the DDG-51 program is outlined in Figure 23. A noteworthy aspect of the 
USN schedule is the exclusive OT-3 period, the results of which support a Fleet 
Introduction decision. 
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ACTIVITIES 



Builders Acceptance ^Post Delivery 

Trials Trials & Trials 



► Shaked own 



MILESTONES 



Final 

Contract 

Trials 



Post 

Shakedown 

Availability 



OT-3 



Ac ceptance 
/Delivery 



Commission 



Fleet 

In troduction 



Figure 23: USN Ship Acceptance Schedule and Milestones 



The Board of Inspection and Survey (INSURV) conducts Acceptance Trials 
and Final Contract Trials for each individual new ship to provide independent 
verification of readiness of ships for acceptance in the USN. Based upon its findings, 
the Board recommends acceptance or final settlement of the contract. The Board 
identifies material conditions which represent departures from the USN General 
Specifications and deficiencies that substantially reduce the ship's fitness for naval 
service or degrade its ability to perform its primary mission.[Ref. 34:p. 1] On 
successful trials completion, NAVSEA accepts the ship from the contractor. 

For comparison, the RAN ship acceptance schedule, as used for the 
Australian Frigates and proposed for the ANZAC frigates, is outlined in Figure 24. 
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Figure 24: RAN Ship Acceptance Schedule and Milestones 



Of note in this schedule is the combined PAT&E / OT&E period supporting 
an AINS decision, and the OT-4 phase leading to OA. OT-4 is not an exclusive T&E 
phase, but is integrated with the normal fleet operations of the ship. AINS is the most 
significant post delivery milestone as it represents acceptance of the ship into the 
RAN and that the project has met it’s responsibilities. 

In the RAN, Delivery (contractual acceptance) of the ship to the 
Government is a contractual matter and is managed by the Project Director. CTE’s 
responsibility is to conduct the Acceptance Trials and report the results. An 
Acceptance Board (AB) is established to provide an assessment of the ship and 
submit recommendations upon which CNS can base his acceptance decision, 
however the AB is quite different from the INSURV Board. The AB is usually 
established for each new ship type and only for first of class. The President, Vice- 
President and Board members serve on the Board part time. The AB’s sole function 
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is to advise CNS on acceptance matters. This is achieved by assessing the adequacy 
of the trials plans, examining the inspection and trials reports, attending trials where 
necessary and then providing an overall assessment of the adequacy of the ship 
against the user requirement. The Acceptance Board does not conduct tests, trials or 
inspections, and is essentially a board of review. 

Acceptance into Naval Service is essentially the transfer/acceptance of the 
ship from the acquisition part of the Navy (ACMAT-N) to the inservice operations 
and support areas, as having met the user requirement. Post delivery trials leading to 
AINS includes the SQT period, with harbour and sea phases. 

For the first of class, it will normally not be possible to trial and evaluate all 
aspects of the operational performance and maintainability prior to AINS. This OT- 
4 period is conducted following AINS and is compiled by RANTAU in consultation 
with the Naval and Maritime Headquarters Staff. It comprises a series of trials and 
evaluations to assess operational effectiveness and operational suitability. [Ref. 
41:p. 15.14] Typically, the Follow-on T&E program will last from 12-24 months, 
depending on the class of ship. At the end of this period RANTAU compiles a report 
in consultation with Maritime Headquarters Staff. The AB assesses the results and 
provides the overview to ensure the program is comprehensive. This assessment 
serves to confirm or modify the assessment at AINS. 

3. Ship Acceptance Summary 

Both Navies now include OT&E in their post-delivery trials of the first of 
class to ensure that the ship is assessed operationally against the User Requirement. 
However the RAN OT-3 period is not true OT&E, but consists mostly of DT&E and 
PAT&E with some OT&E elements. The USN INSURV Board conducts acceptance 
trials on every ship but looks at materiel items only, which is different to the RAN 
AB, which is a board of specialists who review the trials and results of the first of 
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class. OT-3 within the RAN needs to be separated from the DT&E /PAT&E post 
delivery events, and should consist of true OT&E. This would enable the 
Acceptance Board to use its knowledge and experience in assessing the results of 
OT&E against the user requirement, instead of assuming a quality control function 
over what is essentially PAT&E testing. 

G. OT&E PROCEDURES 

This section discusses the procedures adopted by the two Navies in 
implementing OT&E policy. 

1. User Requirement to OT&E Test Plan 

To this concruence between OT&E test documentation and the user 
requirement, both Navies generate a series of documents through the acquisition 
process. Although the documents between Navies have different titles, they have 
similar functions. A comparison of the progression of documentation leading from 
the user requirement to the OT&E test plans is given in Table 18. 

Both Navies can trace a requirement from the capability need through the 
refining requirements documentation to specific a objective in the TEMP, to the test 
plan and ultimately to the final OT&E report 
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TABLE 18 : COMPARISON OF REQUIREMENTS DOCUMENT ATON 



USN 


Document 


RAN 


Document 


Function 


Document 


Mission Need Statement 


Capability 


Defence Force 


(MNS) 


Need 


Capability Proposal (DFCP) 


Operational Requirements 


Refined 


Required Operational 


Document (ORD) 


Requirement 


Characteristics (ROC) 


Top Level Requirement 


Requirement 


Agreed Ships 


(TLR) 


Specifics 


Characteristics (ASC) 


Test and Evaluation 


Overall 


Test and Evaluation 


Master Plan (TEMP) 


T&E Plan 


Master Plan (TEMP) 


OT&E 


OT&E 


Acceptance Management Plan 


Test Plan 


Plan 


(AMP) 



2. Assessment of Operational Effectiveness 

The RAN has found it difficult to define exactly what is required to assess 
operational effectiveness. The RANTAU guidance on the conduct of OT&E [Ref. 
53] recognises that the user requirement is the bench mark for determining the 
degree to which a product is effective. Also that operational effectiveness is best 
assessed by a performance demonstration by normal operating personnel in the 
normal or given environment. But otherwise provides very little guidance on the 
assessment of operational effectiveness. Past techniques employed involve 
analyzing each COI and employing modified Ship Qualification Trial and Fleet 
Exercise Program techniques to evaluate them. Use has also been made of 
simulation, weapon analysis and exercise analysis. The lack of suitable range 
facilities has also limited the assessment of operational effectiveness. 
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The USN, by comparison, has a more effective method of assessing 
operational effectiveness. As do the RAN, the USN examines each COI and related 
operational effectiveness objective, and decides what needs to be known to enable 
each objective to be assessed [Ref. 15:p. 9-1]. They ensure that the appropriate 
environments, threats, etc. are included and that sufficient data will be generated to 
address the COI and objectives. They focus on achieving statistical relevance where 
possible, and making the tests as objective as possible. DOT&E works closely with 
the USN to help determine how many test assets are required to achieve a certain 
degree of confidence that the results are correct. A test may need to be repeated a 
number of times to gain statistical relevance. However, if this is not possible, or the 
direct proving of an ability is not possible (as in the surface warfare example), then 
the test needs to be designed so that what is observed in the test program is projected 
to be what would be observed in reality. [Ref. 19][Ref. 21] These situations require 
expertise and judgement, and compromise between the authorities involved, to 
make the tests as objective as possible. 

3. Assessment of Operational Suitability 

The principles of operational suitability are common between the USN and 
RAN, and similar aspects are assessed. Within the USN, COMOPTEVFOR reviews 
the Integrated Logistic Support Plan (ILSP) against the user requirement. Following 
this review, COMOPTEVFOR determines the suitability tests required based on 
expected reliability, degree of confidence, thresholds, etc. required to be achieved. 
These determine the scope and length of the assessment. Reliability, maintainability 
and availability (RM&A) data is collected and sent to the normal inservice analyst 
of RM&A data, the Naval Weapons Analysis Centre (NWAC) for analysis. 
Specialist analysts within OPTEVFOR design the tests and determine measures of 
suitability. They also evaluate and analyse the adequacy of logistic supportability. 
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However, OPTEVFOR also relies on operational personnel to use their experience 
and knowledge of the system to identify inadequate logistic support. 

The assessment of operational suitability in the RAN has been 
controversial, particularly the assessment of supportability. Before 1987, the 
assessment of operational suitability, and ILS in particular, was rather subjective and 
often controversial. A new approach was taken in 1987 and has continued to be 
refined. The assessment of ILS is now conducted in two phases: 

• ILS Structural Review 

• ILS Operational Review 

The ILS Plan is reviewed and discussions with the functional areas 
handling the support. conducted prior to operational testing. During the operational 
testing period, the ship’s ILS related documentation and records are reviewed, 
maintainability demonstrations are performed and a supportability assessment is 
conducted. The RAN is currently developing a routine in-service RM&A data 
collection and analysis system, however, until that is operational, the data is 
collected and analysed on an ‘as required’ basis by the Trials Authority. 

In summary, the principles of operational suitability assessment are 
common between the USN and RAN. Although assessments by the RAN have 
become less objective and more subjective over recent years, the assessment of 
RM&A is still in its infancy. Both Navies recognise the importance of data 
collection, however, the RAN needs experienced personnel to design tests and 
analyse results. 

4. OT&E Coordination 

The USN recognises the importance of coordination between authorities in 
the successful achievement of T&E. The USN T&E coordinator is responsible to the 
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Director, Test, Evaluation and Technology Requirements (N091). They are 
essentially the OPNAV Staff coordinator for Navy T&E. providing a primary 
contact point for all parties and setting up T&E briefings and meetings. Their 
responsibilities include the chairmanship of the Test and Evaluation Coordination 
Group (TECG) for each major program. [Ref. 37] TECGs are used for complex, 
multifaceted programs which require extensive T&E coordination. Membership of 
a TECG includes the Program Manager, the Sponsor, COMOPTEVFOR, a logistics 
coordinator and others as appropriate (such as a PREINSURV representative). 
TECG recommendations are considered for inclusion in the TEMP. [Ref. 1 l:p. 15] 
Some of the functions of a TECG are the early definition of terms, measures of 
effectiveness and how these are to be measured, and the criteria for acceptable or not 
acceptable. Of note is the formation of a TECG does not imply a joint test team 
approach. Each T&E agency remains fully and solely responsible for conducting 
and reporting the types and phases of T&E for which it is accountable. 

In the RAN, a T&E Planning Group (TEPG) is formed prior to TEMP 
development under the leadership of the T&E manager for the project. Foundation 
members of the TEPG are the Project Director, Project Sponsor, the Design 
Approval Authority and RANTAU, with other parties seconded/called upon as 
required. [Ref. 44:p. 5] The TEPG analyses T&E requirements and estimates 
resources required to meet the requirements. 

The function of these groups is similar, however, the USN TECG is led by 
the Director, Test, Evaluation and Technology Requirements (N091) under the 
CNO, whereas the RAN TEPG is led by each individual Project Director through 
this T&E manager. Despite their best intentions. Project Directors are essentially 
driven by cost and schedule considerations, not the overall T&E adequacy of their 
project. To be truly objective, each TEPG should be chaired by the authority 
responsible for ensuring the adequacy of the RAN’s overall T&E program. 
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5. OT&E Reporting 



Reports provide the OT&E authority’s conclusions regarding a system’s 
operational effectiveness and suitability, and his recommendations regarding the 
systems future, i.e., acceptance, further development, additional OT&E, etc. The 
USN system has three basic principles for OT&E reports: 

• Impartial 

• Complete and Thorough 

• Sole Reporter 

The conduct of OT&E must be impartial and not influenced by the program 
manager or contractors during testing and analysis. COMOPTEVFOR’s 
conclusions are based on a complete and thorough analysis. A report covers a 
complete OT&E phase (e.g., OT-IIIA), relating the test results to the COIs and 
addressing the objectives stated in the TEMP. Quick-look and interim reports are 
usually sent only if the testing could not be completed or when directed by the CNO. 
Although the final report may be a surprise to other authorities. OT&E results and 
analysis are reported through one authority only. Comments from the Commanding 
Officer of the ship under test ship, for example, are sent only to COMOPTEVFOR. 
All operational test data is considered to be the owned by COMOPTEVFOR until 
the final report is signed. 

Within the RAN, the OT&E Authority coordinates the issue of trials 
reports, however, they are issued on completion of each segment of OT&E e.g., o/c 
ASW phase of OT-3 etc. Quicklook reports are also routinely sent on completion of 
each week’s testing. The ship under test also often provides a report, however, it is 
usually forwarded to their operational authority and the Project in addition to the 
Trials Authority. For the results of PAT&E, rapid feedback is necessary to ensure 
timely rectification of defects. However, OT&E results may indicate system 
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deficiencies against user requirements rather than equipment defects, so the results 
need to be fully analysed and the implications assessed before being reported to a 
wider audience. The RAN practice of OT&E reporting probably stems again from 
the lack of delineation between types of T&E and the combined PAT&E / OT&E 
nature of the RAN’s OT-3 testing. 

6. OT&E Personnel Selection and Training 

OT&E is a specialised discipline, with its own philosophy and 
methodology, and so requires a specialist approach with knowledge and experience 
to make it effective. 

a. Selection 

The U.S. community involved with ship OT&E select and conduct 
limited training for their military personnel. Although there are no special selection 
requirements for detailing military personnel for service as Test Directors in 
OPTEVFOR, broad ship operations experience and combat system knowledge are 
preferred. For particular programs, OPTEVFOR make use of subject area experts, 
either resident or borrowed from a non-interested party. Military personnel required 
by DOT&E are selected by background and expertise, preferably war college 
graduates with joint experience. 

No special selection criteria is involved for posting OT&E personnel 
within the RAN, although like the USN, broad ship operations experience and 
combat system knowledge are preferred. Very few personnel have had any T&E 
experience prior to joining the OT&E authority. Also currently the OT&E authority 
is viewed by some as not being career enhancing for warfare personnel. 
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b. Training 

COMOPTEVFOR conducts a four day Operational Test Director’s 
(OTD) overview course covering the major areas of OT&E. Further details of this 
course are included in Appendix A. They also run adhoc segment courses which 
provide acquaints or updates on OT&E subjects e.g., analysis, test plan development 
and threat updates. To assist their personnel in managing OT&E, OPTEVFOR 
publishes the OT&E Director’s Guide which documents their philosophy and 
methodology. [Ref. 21] Other authorities involved with OT&E also send personnel 
to attend this course. Although DOT&E does not conduct courses for new staff, a 
one month handover is usually required. 

The RAN has no training courses on OT&E, or on T&E in general. A 
number of officers have completed the USN Operational Test Directors Course in 
recent years which has improved the knowledge and understanding of OT&E within 
CTE’s organisation considerably. These courses have been arranged on an ad-hoc 
basis through overseas visit submissions rather than as pre-requisite courses for 
particular billets. For guidance of trials personnel and other authorities, CTE has 
developed a number of Standard Operating Procedures addressing certain aspects of 
OT&E. 

c. Career Path 

Although some personnel may develop into OT&E specialists, no 
career path in OT&E is consciously provided in the USN, and no naval personnel 
currently serving in DOT&E have served previously in OPTEVFOR. Similarly in 
the RAN, no career path in T&E is provided. 

To be effective, OT&E ideally requires current, experienced, 
knowledgable personnel with high professional credibility within their field of 
expertise. To achieve this, selection of suitable personnel, adequate training and 
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good guidance documentation is required, and preferably a career path where OT&E 
is seen as suitable career progression by such capable personnel. 

H. OT&E IMPLEMENTATION 

Recent ship projects illustrate how these policies and procedures are 
implemented. The USN DDG-51 program employs a new hull with a new, 
evolutionary combat system. Noteworthy aspects of DDG-51 OT&E included OT- 
II on both combat and propulsion systems LBTS to gain an early indication of 
potential operational effectiveness and suitability which supported the decision to 
approve the DDG-51 to go beyond Low Rate Initial Production. Another 
characteristic of the DDG-51 program was “Whole Ship” OT&E where the whole 
ship is viewed as an integrated warfare system, rather than as a platform for 
individual warfare systems. OT-3 was conducted in a dedicated period following 
CSSQT and Final Contract Trials (but before the workup) to support a Fleet 
Introduction decision. It consisted of fully assessed, free play multi-threat scenarios 
with the emphasis on operational realism. 

The implementation of OT&E policy within the RAN is best demonstrated by 
reviewing examples of ship OT&E conducted to date, and that planned for ships due 
to enter service in the near future. 

The three RAN DDGs underwent a major modernisation and refit over the 
period 1987-1991. The major modernisation occurred with the ships’ combat 
system, command and control system, sensors and weapons, with the aim of 
increasing the ‘supportability’ of the systems rather than increasing the ships’ 
capability. As a low operational risk project, the OT&E followed the established 
RAN policy, however, it was the first to actually implement an OT-4 period to more 
fully assess operational effectiveness and operational suitability leading to OA. 
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Two US FFG-7 class frigates, known as Australian Frigates (AF), are being 
built in Australia. The AF is a low operational risk project, except for the addition 
of the Mulloka sonar. The OT&E followed the established RAN policy, however, it 
failed to plan an OT-4 period to more fully assess operational effectiveness and 
operational suitability of the Mulloka sonar. 

Australia and New Zealand are cooperating in the building of the ANZAC 
frigate. Essentially a MEKO 200 design incorporating a combination of European 
and US equipment, the design is classified as low risk, although modifications to the 
design were necessary to meet RAN requirements. Although only in the early 
stages, the planned OT&E for the ANZAC ship follows the established RAN policy, 
with the OT-3 period combined with the SQT, being conducted prior to AINS. The 
ship class has a higher operational risk than either the modernised DDG or the AF. 

The RAN is building six boats, known as the Collins class, to the Swedish 
Kockums Type 471 design. They will be the first submarines to be constructed in 
Australia and will have a unique equipment fit These submarines possibly have the 
highest operational risk of all current major RAN combatant projects. Consequently 
the increased need for OT&E has been recognised. The planned OT&E program 
demonstrates a departure from the usual RAN OT&E policy, and reflects a growing 
awareness of the need to separate DT and OT activities, and to include a dedicated 
OT-3 period after the usual SQT and ORE. 

These examples of RAN implementation of ship OT&E policy illustrates a 
number of key issues. The operational risk of ship projects within the RAN is 
increasing, leading to a growing awareness for the increased need for ship OT&E. 

Significant differences in OT&E implementation between the USN and RAN 
are, first, that the RAN does not generally conduct OT&E to support the production 
or purchase decision of ships. Second, the RAN has relied on the results of a 
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combined SQT and OT-3 period to provide a basis for AINS, whereas the USN 
conduct a “whole ship" dedicated OT-3 phase to support Fleet Introduction. 

L SUMMARY 

The USN and RAN have similar acquisition systems and similar fundamental 
definitions of T&E. However, the OT&E systems differ in a number of significant 
areas. Both Navies appreciate the importance of T&E in the acquisition process, 
however, the level of importance of T&E, and OT&E in particular, differs greatly. 
The USN recognises that without OT&E, it risks deploying systems which cannot 
adequately perform significant portions of their missions, thus degrading its 
deterrent/defensive capabilities and endangering the safety of military personnel 
who operate and maintain the systems. The importance of OT&E within the RAN is 
not specifically addressed in any T&E policy document. The USN places emphasis 
on OT&E leading to full scale production where the majority of a project’s funding 
is spent. In contrast, the RAN system places a high priority on the post production 
phase. 

The RAN has adapted the more detailed USN T&E definitions to suit its own 
requirements. During this process, the RAN has lost the distinction between the 
types of T&E and often confuses OT&E with PAT&E / DT&E. USN OT&E policy 
has its basis in statutes approved by Congress. The RAN has no such formal basis 
and its OT&E policy is embedded in the more general T&E policy. 

The USN OT&E organisation is well structured with comprehensive 
responsibilities for each participant, and clearly delineated from the DT&E 
structure. Following a number of recent reorganisations, the RAN’s OT&E structure 
is in disarray, with no clear policy maker for T&E in general, or OT&E in particular, 
lacks functional authorities with T&E knowledge to advise Project Directors, and its 
OT&E authority is buried in the in-service area of the Maritime Command. 
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Both Navies have similar policies and procedures for acceptance of ships from 
contractors and then into service. Each Navy uses similar terminology, but these 
have different meanings. OT&E results in the USN is now having a greater input on 
ship acceptance. Although the RAN has traditionally included OT&E in support of 
AINS, the nature of the testing has really been more DT&E / PAT&E rather than true 
OT&E. 

The USN OT&E system has, by necessity, developed into a well organised, 
well documented and effective system. The RAN system, on the other hand, suffers 
from confusing T&E policy documentation, a weak OT&E organisation structure 
and a general lack of OT&E knowledge and appreciation. The determination of the 
OT&E required has traditionally not been achieved early enough in the life of a 
project, and so project funding has not included the provision for OT&E. However, 
OT&E awareness and knowledge within the RAN is growing as evidenced by the 
proposed Collins class post delivery trials. 

The RAN OT&E system, although originally based on USN OT&E philosophy, 
now differs in the importance, interpretation, application and focus of OT&E As a 
result of organisation changes, it now lacks the ability to effectively manage overall 
T&E, let alone OT&E, within the RAN. To achieve efficient and effective trials and 
acceptance of the new higher risk combatants currently under construction, the RAN 
OT&E system needs to be revised to fully support the acquisition process. 
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VI. A MODEL FOR OT&E IN THE RAN 



This chapter proposes a model for the OT&E system in the RAN. The OT&E 
principles derived from the USN system are used to develop a system to suit RAN 
requirements. 

A. TEST & EVALUATION DEFINITIONS 

The basis for the OT&E system are clear definitions of the types and phases of 
T&E. The RAN should continue to recognise the three types of T&E associated with 
ships. The current definitions listed in the CPO Manual [Ref. 38] should be amended 
to better suit the Australian Defence environment. 

1. Development Test and Evaluation (DT&E) 

The DT&E definition remains unchanged as follows: 

DT&E is conducted to assist the engineering design and development process, 
and to verify attainment of technical performance specifications and 
objectives. 

The proposed phases of DT&E are detailed in Table 19. The three phases 



TABLE 19: PROPOSED PHASES OF DT&E 



Current RAN Description 


DT 


Proposed RAN Description 


Validation of design concept. 


DT-1 


Validation of design concept. 


Proving design. 


DT-2 


Demonstration that design meets specifications. 


Demonstration that production 
meets required technical 
characteristics. 


DT-3 


Demonstration that production meets required 
technical characteristics or establish standards for 
first of class. PAT&E is a form of DT-3. 
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are retained, however, the DT-2 definition is amended to clarify the meaning of 
“proving design”, and the DT-3 phase now recognises that PAT&E is a form of 
DT&E. 

2. Production Acceptance Test and Evaluation (PAT&E) 

The definition of PAT&E is revised to recognise that it is conducted not 

only during the contract period, but also during subsequent RAN testing: 

PAT&E is conducted on production items to ensure systems meet technical 
specifications and requirements, and is a type of DT&E. 

The proposed phases of PAT&E are listed in Table 20. Phases PAT-0 to PAT- 
5 remain unchanged. PAT-6 is amended to delete any connection with OT-3, as under 
the proposed model OT-3 will now be conducted separately. The PAT-7 phase is 
deleted as no PAT&E is conducted during the OT-4 phase of testing. 



TABLE 20: PROPOSED PHASES OF PAT&E 



Current RAN Description 


PAT&E 


Proposed RAN Description 


Design and engineering 
development tests 


PAT-0 


Design and engineering 
development tests 


Production and bum-in tests 


PAT-1 


Production and bum-in tests 


Environmental qualification tests 


PAT-2 


Environmental qualification tests 


System development tests 


PAT-3 


System development tests 


Harbour testing 


PAT-4 


Harbour testing 


Sea testing 


PAT-5 


Sea testing 


Operational and Qualification Trials 
(conducted with OT-3) 


PAT-6 


Certification and Qualification 
Trials 


Follow on Operational T&E 
(conducted with OT-4) 


PAT-7 


Nil. 
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3. Operational Test and Evaluation (OT&E) 



The definition of OT&E is revised along USN lines, so that it more 

accurately describes the purpose and nature of OT&E: 

OT&E is conducted to determine a system’s operational effectiveness and 
operational suitability, identify system deficiencies and the need for potential 
modifications to meet established OT thresholds, and develop tactics. OT&E 
has three distinguishing characteristics: 

• It is conducted in an operationally representative environment. 

• It is conducted on production representative equipment using fleet personnel 
for operation and maintenance. 

• It is conducted against a threat-representative simulated enemy carrying out 
threat tactics per the latest threat assessment. 

The proposed phases of OT&E are described in Table 21. These amended 
definitions include reference to the acquisition milestones they support. The 
reference to OPEVAL in the OT-2 phase has been deleted, as the RAN uses the term 
OPEVAL as an abbreviation for an operational evaluation in any OT phase. The 
reference to a prototype in OT-2 has also been deleted as OT-2 may also be 
conducted in a LBTS, or on a sample system if the proposed purchase is "off the 
shelf’. The OT-3 phase is revised to delete the Certification /Qualification Trials as 
they are now recognised as PAT&E. The definition of OT&E includes the use of 
normal fleet personnel so this wording has been deleted from the phase definitions. 
The OT-5 phase is amended to cover all in-service OT&E which could be conducted 
to assess new applications, new tactics, revised threat, etc. Reference to defect 
rectification is deleted also as it is a sub-phase of OT-4, and the assessments of 
modernisations and modifications are deleted as they would be new OT-2 or OT-3 
phases. 
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TABLE 21: PROPOSED PHASES OF OT&E 



Current RAN Description 


OT 


Proposed RAN Description 


Operational assessment of the 
development proposal 


OT-1 


Operational assessment of the 
development proposal to support Full 
Scale Engineering Development 
approval. 


Demonstration of achievement of 
program requirements for operational 
effectiveness and suitability of a 
prototype (OPEVAL) to support 
proceeding to full production 


OT-2 


Demonstration of achievement of 
program requirements for operational 
effectiveness and suitability to support 
a production or purchase decision. 


Demonstration of achievement of 
program requirements for operational 
effectiveness and suitability on 
production of ship/aircraft/system, 
normally in independent operations 
using normal Fleet personnel. 
Certification/Qualification Trials 
include limited reliability, 

maintainability, availability and 
logistic supportability assessments. 


OT-3 


Demonstration of achievement of 
program requirements for operational 
effectiveness and operational 
suitability on production of ship/ 
aircraft/system to support Acceptance 
into Naval Service. Includes limited 
reliability, maintainability, availability 
and logistic supportability 
assessments. 


Demonstration of achievement of 
program requirements for operational 
effectiveness and operational 
suitability on production of ship/ 
aircraft/system using normal Fleet 
personnel in a multi-force, multi-threat 
environment. Includes detailed 
reliability, maintainability, availability 
and logistic supportability 

assessments. 


OT-4 


Demonstration of achievement of 
program requirements for operational 
effectiveness and operational 
suitability on production of ship/ 
aircraft/system in a multi-force, multi- 
threat environment to support 
Operational Acceptance. Includes 
detailed reliabilitv, maintainabilitv, 
availability and logistic supportability 
assessments. 


Follow-on OT for assessment after 
modernisations, new applications or 
defect rectifications after OT-4. 


OT-5 


In-service OT&E, which could include 
new applications, new tactics, revised 
threat, etc. 
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4. T&E Types in Total Ship Test Program 

The Total Ship Test Program (TSTP) concept is now well established in 
RAN shipbuilding projects, although modified from the U.S. system. These refined 
T&E phases can be incorporated into an amended TSTP. However, instead of being 
grouped together as in the current TSTP, the DT&E and OT&E events are separate 
categories of testing. Because these separate DT&E, PAT&E and OT&E events are 
integrated within the TSTP, these trials are termed collectively as “Integrated Tests 
and Trials”, which is an extension of the current RAN application of this term. The 
proposed categories of ITT are listed in Table 22. 



TABLE 22: CATEGORIES OF ITT 



ITT Part 


ITT Category 


T&E T>pe 


Description 


ITT Part 1 


ITT-O* 


PAT-0 


Design and engineering development tests 




ITT-1 


PAT-1 


Production and bum-in tests 




ITT -2 * 


PAT-2 


Environmental qualification tests 




ITT-3 * 


PAT-3 


System development tests 




ITT A 


PAT-4 


Harbour testing 




ITT -5 


PAT-5 


Sea testing 


ITT Part 2 


ITT-6 


DT-3 / PAT-6 


Certification and Qualification Trials 




ITT-7* 


OT -3 


Operational T&E 


ITT Part 3 


ITT -8 * 


OTA 


Follow on Operational T&E 



Note: * Only performed on first of class 



The Certification and Qualification trials (ITT-6) are classified as DT&E for 
the first of class only, as standards of performance are established. For follow-on 
ships these trials are classed as PAT&E, as the standards established by the first of 
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class become the benchmark to be achieved by follow-on ships. 1TT-0, ITT-2, ITT- 
3 and ITT-7 tests are performed only on the first of class. ITT-8 testing would only 
be performed once for a ship class, but may be conducted progressively on any ship 
of the class. 

B. T&E IN THE ACQUISITION PROCESS 

To ensure the progressive assessment of operational effectiveness and 
operational suitability during the acquisition process, OT&E is scheduled towards 
the end of each acquisition phase, usually after the DT phase. The proposed T&E 
schedule is shown in Figure 25. No change is suggested to the current relationship 
between OT&E phases and acquisition milestones, however, the figure separates 
DT&E/PAT&E from OT&E to clearly differentiate these different types of testing. 




Figure 25: Proposed Test and Evaluation in the Acquisition Process 
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As the construction of proven overseas ship designs adapted to RAN 
requirements, does not require the conduct of OT-2 prior to a production or purchase 
decision, the first opportunity to evaluate potential operational effectiveness and 
operational suitability may be in a LBTS after the production decision has been 
made. To ensure consistency of terminology, this testing should not be termed OT- 
2 as it does not support a production or purchase decision. It should be classified as 
the first phase of OT-3. 

C. RAN OT&E ORGANISATION 

The objective of an OT&E organisation structure is to provide effective OT&E 
to support the acquisition process, and hence the combat effectiveness of the RAN. 
The proposed OT&E structure is shown in Figure 26. The effectiveness of OT&E 




Figure 26: Proposed RAN OT&E Organisation 



176 



can be achieved by devoting appropriate resources of both personnel and funding, 
by considering the requirements of all functional areas concerned with OT&E, and 
by coordinating these different functions. 

1. Policy Maker and Overseer 

An authority is required to formulate OT&E policy, and to evaluate and 
oversee the OT&E process. OT&E policy should come from an operational, not 
technical, functional area. HQADF now contains the centralised force development 
and user requirements organisation. As OT&E is the feedback of the performance of 
the final product against the user requirement, then ideally, the head policy maker 
and overseer of OT&E should be within HQADF or DoD, along the lines of the U.S. 
system. 

A shorter term, RAN-only, solution would be to include the overseer and 
policy maker function within the revised Director General Naval Policy and Warfare 
(DGNPW) organisation under DCNS. Although this area has recently been reduced, 
DGNPW still retains the responsibility for naval warfare information and expertise. 
The Director of Naval Warfare (DNW), under DGNPW, provides an information 
brokerage service to senior level management. He currently has a secondary 
function of providing naval warfare input to the Defence Capability Proposals 
developed by HQADF, and providing advice on naval warfare operations, 
equipment and personnel aspects. [Ref. 55:p. A-4] The responsibility of providing 
advice and input on OT&E matters, as part of the OT&E policy maker and overseer 
function for the RAN, is seen as a natural extension to these current responsibilities. 

2. Sponsor 

The Sponsor of major naval acquisitions is DGFD(Sea) within HQADF. 
Traditionally in the RAN, it has been the Sponsor’s responsibility to initially 
determine the T&E requirements in an acquisition program and to detail those 
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requirements in the capability proposal. To be able to determine the appropriate 
nature and extent of OT&E required, the Sponsor needs access to OT&E knowledge 
and experience. A close relationship needs to be maintained by DGFD(Sea) with the 
OT&E policy maker and the OT&E authority (Note dotted line). 

3. Developing Agency 

The Materiel Division under ACMAT-N, performs the Developing Agency 
function within the RAN. Currently major combatant projects establish a T&E 
manager, with the larger ones also establishing a manager exclusively for post 
delivery trials. For example, the Australian Frigate Project has a Post Delivery Test 
and Trials manager, and the New Submarine Project has just established a 
Submarine Project Transition and AINS manager. However, ACMAT-N does not 
have a central T&E policy authority to advise these T&E managers. Although the 
Director of Naval Equipment Production (DNEP) has responsibility for providing 
specialist technical advice on technical/engineering activities, which includes trials 
and acceptance of equipment [Ref. 64], he specialises more in advice for contract 
management than for post delivery T&E. 

To assist all RAN projects with T&E matters, an authority is required 
within the Materiel Division whose responsibilities would include assisting Project 
Directors in complying with policies, incorporating lessons learned from previous 
projects and reviewing the T&E aspects of projects at major milestones. Although 
primarily a DT&E policy authority, this office would also have an input into the 
OT&E process by reviewing the TEMP, and by advising projects how to prepare for 
OT&E. The Director General Naval Engineering Requirements (DGNER) is 
currently responsible for the design, development, acceptance into service and 
through life support of ships. This engineering policy directorate appears to be well 
suited to the DT&E policy and review function. 
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4. OT&E Tester & Evaluator 



Ideally, the OT&E Tester & Evaluator should be independent and focus 
exclusively on OT&E. He should also have operational credibility and influence in 
operational matters relating to OT&E. The current OT&E Tester and Evaluator, 
Commander, Test and Evaluation (CTE), within the Maritime Command is 
independent from developing and production agencies, but is responsible to the end 
user of the system. This arrangement has advantages to CTE of access to 
knowledgeable and experienced fleet staff warfare officers and fleet planners for 
assistance in trials planning and assessment. However, there are possible 
disadvantages to this lack of independence from the end user. The Maritime 
Commander may have aims and objectives which could conflict with the total 
impartiality of OT&E reporting. As the Maritime Commander establishes 
scheduling priorities for his resources, priority given to OT&E is subject to the 
politics of requirements and funding. OT&E may take a low priority behind 
operations and training. The Maritime Commander may also not wish to “stress” the 
system during OT&E, in case he perceives that operational deficiencies so 
highlighted may be seen as having some reflection on his own organisation. 

CTE is currently multifunctional, conducting DT&E and PAT&E in 
addition to OT&E. This results in a lack of differentiation within CTE's organisation 
between DT&E/PAT&E and OT&E. Also being placed under the inservice 
engineering area of the Maritime Command, he lacks visibility and influence within 
the warfare community and hence credibility in operational matters critical to 
OT&E. CTE’s OT&E responsibilities are overshadowed by his DT&E/PAT&E 
duties. 

To achieve the advantages of DT&E and OT&E separation, a revised 
organisation is proposed, similar to that outlined by the former RANTAU in 
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Reference 63. All CTE’s current DT&E/PAT&E responsibilities, and those already 
the responsibility of CSO(E), would remain with an in-service trials team under 
CSO(E). This trials team, headed by an engineering branch officer, would be formed 
from the current CTE technical personnel, and could also conduct certifications. 
Light Off Examinations, and the technical aspects of Ship Qualification Trials. The 
team would provide technical assistance and advice to a separate OT&E trials team. 

CTE’s OT&E duties would be removed and managed by a separate group. 
Commander, OT&E (COT&E), working under an operational authority. There are 
two such authorities within the Maritime Command; the Chief Staff Officer 
(Operations) (CSO(O)) who is responsible for plans and operations, and 
Commodore Flotillas (COMFLOT) who is responsible for Fleet readiness. As 
OT&E is the assessment of operational effectiveness and suitability, COT&E should 
be responsible to COMFLOT. Headed by a Warfare branch officer, COT&E would 
manage the post delivery trials period for all ships from the MHQ perspective. He 
would still manage the SQT, which, although not true OT&E, still has some 
operational input which could not be met by the in-service trials team under 
CSO(E). He would be the OT&E authority with responsibilities to manage all RAN 
OT&E including OT-3, which is now only conducted on first of class. 

5. OT&E Organisation Summary 

This proposed reorganisation achieves the aim of DT&E/OT&E separation 
with some concessions necessary due to manpower constraints. Also, by including 
OT&E under COMFLOT and managed by a Warfare Officer, OT&E could achieve 
the visibility, credibility and influence necessary within the RAN. 
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D. RAN OT&E POLICY IMPLEMENTATION 



1. OT&E Policy Documentation 

RAN T&E documentation needs to be clear, concise and comprehensive so 
that is serves as a valid reference and guide for all personnel involved with T&E. To 
correct the current inconsistent and out of date documentation, a major rewrite, 
rather than just updating is required. This rewrite of T&E documentation could be 
the vehicle by which the T&E system within the RAN is overhauled. Before the 
documentation is revised, T&E definitions and terminology need to agreed within 
the RAN and DoD. Then the OT&E and DT&E organisations within the RAN need 
to be resolved, and the appropriate policies decided. This could be achieved by the 
development of a revised RAN T&E DI(N), followed by ABR 1921which would be 
firmly based on the T&E DI(N). 

2. Independence of OT&E Agency 

Given that the proposed COT&E is organisationally independent from the 
developing and production agencies, his procedural independence should also be 
achieved to ensure impartiality and validity of testing. When OT&E is conducted, it 
should not be influenced by the Project or DAA. No Project or DAA personnel 
should be present during testing. A final report only should be issued by the 
Maritime Commander, with no progress or interim report (unless the testing is 
unusually lengthy), to allow full analysis and evaluation of the test results. This 
applies to OT&E conduct and reporting only. As the SQT period is classified as 
PAT&E it should continue to be conducted with the cooperation and assistance of 
Project authorities. Similarly, to ensure test validity and impartiality, contractors 
should not be present during OT&E. 
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3. T&E Planning 



The RAN has embraced the U.S. TEMP concept as the single executive 
document for the management of T&E for major acquisitions. Although required 
early in a project, recent TEMPs have not been raised until after project funding has 
been approved. TEMPS need to be approved earlier to enable inclusion of post 
delivery PAT&E/DT&E and OT-3 in the Project Budget, and OT-4 in MHQ budget. 
The authority for DCNS to release the TEMP was based on his role as Project 
Sponsor. With HQADF now assuming this role, the DCNS function in TEMP 
development is unclear. The TEMP review process should be reviewed as part of the 
proposed OT&E system. 

A T&E Planning Group (TEPG) is currently formed prior to TEMP 
development under the leadership of the T&E manager for the project to analyse 
T&E requirements and estimate resources required. The TEPG is similar to the USN 
TECG with one significant difference. The USN TECG is led by the Director, Test, 
Evaluation and Technology Requirements (N091) under the CNO, whereas the 
RAN TEPG is led by each individual Project Director through this T&E manager. 
Despite their best intentions. Project Directors are essentially driven by cost and 
schedule considerations, not the overall T&E adequacy of their project. To be truly 
objective, each TEPG should be chaired by the authority responsible for ensuring 
the adequacy of the RAN’s overall T&E program. The proposed RAN OT&E policy 
maker and overseer, DGNPW’s organisation, may best be suited to the chairmanship 
of each TEPG. 

4. Combined DT&E and OT&E 

With the recognition of the differences between DT&E/PAT&E and OT&E 
the RAN can now appreciate the advantages and difficulties of combined DT&E/ 
PAT&E and OT&E. By classifying SQT as PAT&E, and conducting a separate 0T- 
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3 period for first of class, the conduct of combined and/or concurrent DT&E/PAT&E 
and OT&E can be addressed on the requirements of each individual trial. 

5. Foreign Weapons Evaluation 

Despite the most recent Australian Government "White Paper” on Defence 
Policy [Ref. 6:p. 70] requiring the need to be able to determine the performance in 
the Australian environment of equipment of both overseas and local origin, no DoD 
or RAN T&E policy specifically addresses the evaluation of foreign systems. The 
proposed revision of the OT&E system, and consequent DT&E revision, would 
result in a more credible and useful organisation. Policies could then be developed 
regarding OT&E and DT&E in the evaluation of foreign weapons systems. The 
current process of evaluations of foreign weapons by adhoc Project Teams could be 
modified to include greater formal involvement by T&E authorities. 

6. Land Based Test Sites 

LBTS can be used to highlight problems of operational effectiveness and 
suitability early in system development. Although LBTS are part of both recent 
major RAN combatant projects, the conduct of OT&E in these sites is planned to be 
limited to a non-interference basis with system development and integration. OT&E 
needs to be conducted during a dedicated period in the LBTS following successful 
DT&E, to gain an estimate of the potential operational effectiveness and suitability 
of the system. Again, the need for early planning to include OT&E in schedules and 
funding is highlighted. 

7. Whole Ship Testing 

The current OT&E to support an Acceptance into Naval Service (AINS) 
decision (OT-3) is little more than an extended SQT period where DT&E/FAT&E is 
conducted with some operational assessment. It is not considered to be adequate to 
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assess the first of a new ship class against the user requirement. To assess the 
operational effectiveness and suitability of a ship against the user requirement with 
the degree of confidence required for the Acceptance Board to support an AINS 
decision, a dedicated, free play, scenario based, OT-3 period is required. The 
proposed Delivery to AINS trials schedule, which is based on that currently 
proposed for HMAS COLLINS is shown in Figure 27. 




Figure 27: Proposed Delivery to AINS Trials Schedule 



OT-3 should be conducted after the ship has completed the SQT and 
Operational Readiness Evaluation to ensure that the testing is conducted on a 
worked up and materially proven ship. To be effective the testing should be 
conducted of the overall ship as a complete weapons system. The recent USN 
innovation of “whole ship testing” provides a good example of the testing proposed. 
Whole ship testing includes engineering and technical problems combined with 
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combat system activity. It emphasises operational realism by testing the ship as 
though in war, and is designed to provide a realistic threat in an operational 
environment, and an element of surprise to provide realistic tests to support 
completion of the OT&E objectives and resolutions of the COIs. 

The OT-4 period leading to Operational Acceptance (OA) proposed in this 
OT&E model is also based on the schedule currently proposed by CTE for HMAS 
COLLINS. An example of an AINS to OA trials schedule is shown in Figure 28. 




Figure 28: Proposed AINS to OA Trials Schedule 



It includes a long term RM&A and ILS evaluation, further testing to 
achieve statistically significant results in critical areas and the assessment of the 
operational effectiveness and operational suitability of the ship in a major exercise 
and on deployment. 
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E. RAN SHIP ACCEPTANCE 

The Acceptance Board, with its broader experience and knowledge than Trials 
Authority, is generally agreed within the RAN as being successful as a board of 
review of the planning and results of T&E for individual projects, and 
recommending PA and AINS. As an ad-hoc organisation consisting of various 
functional specialists brought together part time, however, the AB suffers from a 
number of problems. 

The AB requires guidance as to its role and functions, T&E philosophy and the 
procedures of the acceptance process. Although the AB attempts to obtain sufficient 
guidance from the T&E documentation, it has found this documentation to be 
unclear and conflicting. To enable AB consistency and development, the Board 
needs an administrator, or permanent VP. RANTAU was the administrator and 
provided guidance to the Board, with CTE now assuming this role. However, the 
AB should be independent from the T&E authority as it assesses the trials planned 
and conducted by that authority. 

With only DT&E/PAT&E and the minimal operational aspects of the previous 
OT-3 testing conducted by the RAN, the AB have tended to focus on materiel 
deficiencies. With a more comprehensive OT-3 period proposed to be conducted, 
the AB members could more effectively use their knowledge and experience in 
assessing the results of OT&E against the user requirement rather than on 
workmanship and materiel deficiencies. 

The AB should be convened for the first of class only, providing the specialist 
expertise to interpret requirements and determine the level of acceptability required 
for AINS. The AB consists of Naval Officers drawn part-time from their normal 
jobs, and so the cost appears to be minimal. The AB, however, has a high 
opportunity cost, which is the value of the next best alternative on which the 
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members could be working instead of serving on the Board. Once the first of class 
is accepted the Board’s task is complete. The interpretations and levels of 
acceptability established by the AB then guide the trials authority in recommending 
acceptance of each of the remaining ships. 

F. RAN OT&E PROCEDURES 

1. Assessment of Operational Effectiveness 

With the proposed inclusion of a dedicated OT-3 period, the OT&E 
authority will need to develop tests for a free play, scenario based environment 
specifically designed to assess the COIs against the User Requirements. This could 
be achieved along USN lines where each COI and its related operational 
effectiveness objective is examined and the information needed to be known to 
enable each objective to be assessed is determined. The test design would have to 
take into consideration data collection methods and range considerations. The 
analysis of these tests may be different from the weapons analysis currently 
performed. Current weapons analysis focuses on individual weapon system 
performance, usually from “fire push” to “weapon destruct”, whereas whole ship 
testing in a free play environment would require a broader analysis. An increased of 
use modeling & simulation may also be required to assess some systems in a cost 
effective manner. It is beyond the scope of this thesis to investigate these 
requirements, however, close liaison between the OT&E authority and the RAN’s 
current analysis groups will be required to achieve valid testing and subsequent 
analysis. 

2. Assessment of Operational Suitability 

The current operational suitability areas now assessed are considered valid. 
These are listed in Table 23. 
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TABLE 23:SUITABILITY ISSUES FOR ASSESSMENT 



Suitability Issues 

Availability 

Reliability 

Maintainability 

Safety 

Human Factors 
Interoperability 
Compatibility 

Integrated Logistic Support including: 

Maintenance Planning 

Supply Support 

Support and Test Equipment 

Technical Data and Documentation 

Manpower and Personnel 

Training and Training Devices 

Facilities 

Packaging, Handling, Storage and Transport 
Computer Support 
Configuration Control 



The assessment of operational suitability, especially for the RAN's current 
combatant projects, should focus on interoperability and compatibility as these 
present the highest risks with the integration of proven equipment in a unique 
combination in a ship platform. 

The assessment of ILS should continue to be conducted in two phases: 

• ILS Structural Review. Conducted prior to operational testing, this review 
assesses if the ILS intended to provided will support the ILS policy of the 
Project. 

• ILS Operational Review. During the operational testing period, the ship's 
ILS related documentation and records are reviewed, maintainability 
demonstrations are performed and a supportability assessment is conducted. 
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RM&A data is collected and analysed during OT-3 to give an indication of 
performance prior to AINS, and then continued for 12 months during the OT4 
phase. The data should continue to be collected and analysed on an “as required”, 
vice continuous, basis by the Trials Authority until the RAN develops its routine in- 
service RM&A data collection and analysis system. The analysis of RM&A data is 
a specialist task, currently outside the experience of most uniformed personnel. To 
achieve significant and valid results, an RM&A analysis capability needs to be 
established, possibly within the Maritime Command’s weapons assessing area. 

3. OT&E Personnel, Training and Career Management 

An effective OT&E system is dependent on knowledgable and experienced 
personnel. OT&E is a specialised discipline, with its own philosophy and 
methodology, however, very few personnel have had any T&E experience 
knowledge prior to joining the OT&E authority. This makes training important, but 
the RAN has no training courses on OT&E, or on T&E in general. A number of 
officers have completed the USN Operational Test Directors Course in recent years 
which has proved beneficial to the knowledge and understanding of OT&E within 
CTE. This course should be a pre-requisite for key billets within the OT&E authority 
until the RAN develops enough knowledge and expertise to develop its own course. 
Although the USN OTD course is only of a weeks’ duration, it is understood that the 
USN is investigating a more comprehensive course. Other authorities also require 
training in OT&E. The OT&E authority needs to develop acquaint courses for other 
personnel with a need to understand OT&E e.g.. Project T&E managers and 
capability requirements developers. Also to increase the awareness of OT&E in the 
RAN, the OT&E authority needs to deliver presentations regularly to Project 
Directors Courses, the ILS Management and Acquisition Course (ILSMAC), 
Defence management courses, and other appropriate acquisition courses. 
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Although some personnel may develop into OT&E specialists, no career 
path in OT&E is provided in the RAN. The warfare officer functional heads within 
the OT&E authority cannot currently aspire to command of the authority as that 
position is currently a Weapons Electrical Engineer’s billet. The proposed change of 
the head of the OT&E authority to a Warfare Officer as COT&E should provide a 
career path within the authority. As the credibility of OT&E increases, the 
knowledge and experience gained by warfare officers in this area will be seen to be 
of more benefit in future warfare related positions, thus enabling a longer term career 
path. The proposed employment of COT&E under COMFLOT should also be more 
career enhancing for operations personnel than the current arrangement under 
CSO(E). 

The professionalism and knowledge of T&E personnel could be enhanced 
by membership in a related professional organisation. The International Test and 
Evaluation Association (ITEA) is one such professional organisation. 
Headquartered in the U.S. with a chapter in Australia, the association encourages the 
development and exchange of technical information in the field of test and 
evaluation. The corporate membership of a suitable professional organisation may 
be worthwhile to the RAN. 

G. OT&E MODEL SUMMARY 

The basis for this OT&E model are clear definitions of the types and phases of 
T&E. The current definitions are amended and phases of T&E are refined to better 
suit the Australian Defence environment. These refined T&E phases are 
incorporated into an amended TSTP, termed collectively as “Integrated Tests and 
Trials”, which recognises the delineation between DT&E and OT&E. No change is 
suggested to the current relationship between OT&E phases and acquisition 
milestones. 
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The current OT&E organisation structure is modified to enable effective OT&E 
to support the acquisition process. It provides for authorities responsible for the 
management, policy making and oversight, and specification of OT&E. This 
proposed reorganisation closely achieves the aim of DT&E/OT&E separation. 

The model proposes the inclusion of a dedicated OT-3 period in a free play, 
scenario based environment. The assessment of operational effectiveness is 
specifically designed to assess the COIs against the User Requirements and will 
require a different approach to analysis. The assessment of operational suitability 
focuses on interoperability and compatibility, and the assessment of ILS is achieved 
using current methods, but with improved RM&A analysis. This enhanced OT&E 
and minor modifications to the AB should provide CNS with a more comprehensive 
and thorough recommendation for AINS. 

To be effective, OT&E ideally requires current, experienced, knowledgable 
personnel with high professional credibility within their field of expertise. To 
achieve this, the model includes adequate training, good guidance documentation, 
and a career path where OT&E is seen as career progression by capable personnel. 

This OT&E model for the RAN uses principles derived from the USN system. 
By taking consideration of resource limitations, characteristics and culture of the 
RAN, it establishes an effective OT&E organisation which is important to the 
testing, evaluation and subsequent acceptance decision of the new combatants. 
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VII. CONCLUSIONS 



A. GENERAL CONCLUSION 

The USN OT&E system has, by necessity, developed into a well organised, 
well documented and effective, if complex, system. The RAN system, on the other 
hand, suffers from conflicting and confusing T&E policy documentation, a weak 
OT&E organisation structure and a general lack of OT&E knowledge and 
appreciation. The determination of OT&E required has traditionally not been 
achieved early enough in the life of a project, and so project funding has not included 
the provision for OT&E. The RAN OT&E system, although originally based on 
USN OT&E philosophy, now differs in the importance, interpretation, application 
and focus of OT&E. As a result of organisation changes, it now lacks the ability to 
effectively manage overall T&E, let alone OT&E, at a time when the operational 
risk of ship projects is increasing. To achieve efficient and effective trials and 
acceptance of the new higher risk warships currently under construction, the RAN 
OT&E system needs to be revised. 

B. SPECIFIC CONCLUSIONS 

1. OT&E in the Acquisition Process 

Australia’s Defence strategy has progressed from a position of dependence 
on allies, to a positive acceptance of both self-reliance and regional influence. To 
achieve its Defence goals, however, Australia devotes much less of its resources and 
places less priority on R&D than does the U.S. The need for local T&E is recognised 
by the Australian Government and DoD, for both indigenous and overseas systems, 
however, the importance of OT&E in particular, is not specifically addressed in any 
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Australian T&E policy document. This lack of recognition of the importance of 
OT&E is perhaps due to the RAN procuring low risk, complete ships in the past 
(e.g., FFG-7 class) with the wealth of USN DT&E and OT&E behind it. 

U.S. OT&E is viewed as being more important as a basis for a decision to 
proceed beyond Low Rate Initial Production, than on the introduction into service 
of the final production item. This is because the “big bucks” of most Defence 
acquisitions are spent during the production phase. OT&E, however, has started to 
play an important role also in the assessment of first of class systems for introduction 
into the USN Fleet. 

2. OT&E Organisation 

The RAN and Australian DoD have reorganised over the past five years to 
create a more effective and efficient user requirements organisation, and to better 
accord with Program Management and Budgeting principles. However, as a result, 
the RAN has no T&E, or OT&E policy maker and overseer, no OT&E coordinator 
within Navy Office, and the RAN’s OT&E authority is buried within the in-service 
engineering management area of the Maritime Command. 

3, OT&E Policy 

Where USN OT&E policy has its basis in the statutes approved by 
Congress, the RAN has no such formal basis and its OT&E policy is embedded in 
the more general T&E policy. The RAN has adapted the more detailed USN T&E 
definitions to suit its own requirements. During this process, however, the RAN lost 
the distinction between the types of T&E, and often confuses OT&E with PAT&E7 
DT&E. Overall, RAN T&E documentation is not comprehensive, inconsistent and 
fails to provide adequate guidance. 

The Commander, Test and Evaluation (CTE), on the staff of the Maritime 
Commander, is independent from the development and production agencies, but not 
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from the end user, as in the USN. The implications of this lack of independence is 
that the end user may have his own aims and objectives which conflict with the total 
impartiality of OT&E conduct and reporting. 

Despite the most recent Australian Government '‘White Paper” on Defence 
Policy requiring the need to be able to determine the performance in the Australian 
environment of equipment of both overseas and local origin, no DoD or RAN T&E 
policy specifically addresses the evaluation of foreign systems. 

Although used primarily for the purpose of conducting DT&E, the USN has 
demonstrated that early OT&E on Land Based Test Sites (LBTS) can give an 
estimation of potential operational effectiveness and suitability, and hence identify 
potential operational problems early and minimise operational risk. Although LBTS 
are being developed for both the ANZAC frigate and Collins submarine projects, the 
use of these LBTS for early OT-3 was not planned in the original project schedules, 
and so is now subject to them being used on a non-interfering basis to the contractor. 

The RAN does not clearly differentiate between the types of T&E, which 
leads to DT&E and OT&E often being conducted over the same period, by the same 
test team. Although the RAN recognises, by definition, the difference in objectives 
and methodology between the types of T&E, no limitations or guidance as to the 
possible hazards of this combined testing approach are addressed. 

4. Ship Acceptance 

As an ad-hoc organisation consisting of various functional specialists 
brought together part time, the Acceptance Board (AB) suffers from a number of 
problems. It requires guidance as to its role and functions, T&E philosophy and the 
procedures of the acceptance process. Although the AB attempts to obtain sufficient 
guidance from the T&E documentation, it has found this documentation to be 
unclear and conflicting. With only DT&E/PAT&E and the minimal operational 
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aspects of the OT-3 testing conducted by the RAN, the AB has tended to focus on 
materiel deficiencies, rather than the performance of the first of class against the user 
requirement. 

5. OT&E Procedures 

The RAN recognises that the user requirement is the bench mark for 
determining the degree to which a system is effective, and also that operational 
effectiveness is best assessed by a performance demonstration by normal operating 
personnel in the normal or given environment. Assessing operational effectiveness 
has involved analyzing each Critical Operational Issue and then employing modified 
Ship Qualification Trial or Fleet Exercise Program techniques to evaluate them. The 
RAN has very little guidance on the assessment of operational effectiveness. 

The principles of operational suitability assessment are common between 
the USN and RAN. Although assessments by the RAN have become more 
subjective over recent years, the assessment of RM&A is still in its infancy. The 
RAN requires experienced personnel to design suitability tests and analyse results. 

The authority for the Deputy Chief of Naval Staff (DCNS) to release the 
Test and Evaluation Master Plan (TEMP) was based on his role as Project Sponsor. 
Now with HQADF assuming this role, the DCNS function in TEMP development is 
unclear. The TEMP approval process requires review. 

The RAN Test and Evaluation Planning Group (TEPG) for each major 
project is chaired by each individual Project Director through his T&E manager. 
Despite their best intentions. Project Directors are essentially driven by cost and 
schedule considerations, not the overall T&E adequacy of their project. To be truly 
objective, each TEPG should be chaired by the authority responsible for ensuring 
the adequacy of the RAN’s overall T&E program. 
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The early dissemination of OT&E results, before a full analysis is complete 
and the implications assessed, can lead to other authorities taking hasty action on 
incomplete information, and can prejudice the OT&E authority’s final conclusions 
and recommendations. OT&E results should indicate system deficiencies against 
user requirements rather than equipment defects, so the results need to be fully 
analysed and the implications assessed before being reported to a wider audience. 

Within the RAN, the current OT&E to support an AINS decision (OT-3) is 
little more than an extended SQT period where DT&E/PAT&E is conducted with 
some operational assessment. It is not considered to be adequate to assess the first 
of a new ship class against the user requirement. 

To be effective, OT&E requires experienced and knowledgable personnel 
with high professional credibility within their field of expertise. To achieve this 
requires selection of suitable personnel, adequate training and good guidance 
documentation, and preferably a career path where serving in the OT&E Authority 
is seen as career progression by such capable personnel. 
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VIII. RECOMMENDATIONS 



The OT&E model recommended for the RAN uses principles derived from the 
USN system. By taking consideration of resource limitations, and the characteristics 
and culture of the RAN, it establishes an effective OT&E organisation which is 
important to the testing, evaluation and subsequent acceptance decision of the new 
combatants. 

A. GENERAL RECOMMENDATIONS 

It is recommended that the proposed OT&E model be reviewed, evaluated and 
implemented in the RAN to achieve an effective OT&E system to support the 
introduction of operationally effective and operationally suitable new combatants 
into the RAN. 

B. SPECIFIC RECOMMENDATIONS 

1. OT&E Definitions 

It is recommended that: 

a, the current definitions and phases of ship T&E be amended to better suit 
the Australian Defence environment. 

b. the current TSTP be revised to show DT&E and OT&E events as 
separate categories of testing, and these trials be termed collectively as 
Integrated Tests and Trials (ITT). 
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2. OT&E Documentation 

It is recommended that: 

a. the RAN T&E DI(N) and ABR 1921 be revised to achieve 
comprehensive and consistent T&E policy guidance. 

3. OT&E in the Acquisition Process 

It is recommended that: 

a. OT&E be conducted, where possible, prior to the production or 
purchase decision in the acquisition process to minimise risk. 

b. OT-3 be separated from the DT&E /PAT&E post delivery trials. 

c. OT-3 be conducted as a dedicated, free play, scenario based period after 
the completion of SQT and ORE to ensure testing is conducted on a 
worked up and materially proven ship. 

d. OT-3 of the overall ship be conducted as a complete weapons system. 

4. OT&E Organisation 

It is recommended that: 

a. the appointment of a DoD OT&E policy maker and overseer within 
HQADF be investigated. 

b. DGNPW be appointed as the OT&E policy maker and overseer within 
the RAN. 

c. a close working relationship on T&E matters be required between 
DGFD(Sea), DGNPW and the OT&E authority. 

d. DGNER be appointed as the Materiel Division T&E agency to assist 
Project Directors in complying with policies, incorporate lessons 
learned from previous projects, review the T&E aspects of projects at 
major milestones, and advise projects how to prepare for OT&E. 
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e. the current CTE organisation be disestablished. 

f. an “In-service Trials” team be established under CSO(E) to conduct 
CTE's current DT&E/PAT&E responsibilities, and those already the 
responsibility of CSO(E). 

g. Commander, OT&E (COT&E) be established as the RAN's OT&E 
authority under COMFLOT, to manage OT&E and the post delivery 
trials period for all ships from the MHQ perspective. 

5. OT&E Policy 

It is recommended that: 

a. policy be developed to enable the performance in the Australian 
environment of foreign equipment to be evaluated efficiently, 
effectively and consistently. 

b. the TEMP approval process be reviewed to ensure appropriate 
authorities are consulted and approve the TEMP. 

c. the TEPG be chaired by DGNPW as the authority appointed to ensure 
the adequacy of the RAN’s overall T&E program. 

d. OT&E not be influenced by the Project or DAA. and that Project or 
DAA personnel not be present during testing. 

e. contractors not be present during OT&E. 

f. TEMPs for non-development projects be approved before production 
approval to enable inclusion of post delivery PAT&E/DT&E and OT-3 
in the Project Budget, and OT-4 in MHQ budget. 

g. OT&E be conducted during a dedicated period in the LBTS following 
successful DT&E, to gain an estimate of the potential operational 
effectiveness and suitability of the system. 



199 



6. Ship Acceptance 

It is recommended that: 

a. the Acceptance Board remain as a board of review. 

b. the Acceptance Board be convened for the first of class only. 

7. OT&E Procedures 

It is recommended that: 

a. The OT&E authority develop tests for a free play, scenario based 
environment specifically designed to assess the COIs against the user 
requirement. 

b. analysis requirements of these tests be assessed, as they may be 
different from the weapons analysis currently performed. 

c. experienced personnel assist in designing suitability tests and analysing 
results. 

d. interoperability and compatibility be a major focus during the 
assessment of operational suitability, especially for the RAM’s current 
combatant projects. 

e. an improved RM&A analysis capability be established. 

f. RM&A data continue to be collected and analysed on an ‘as required’ 
basis until the RAN develops its routine in-service RM&A data 
collection and analysis system 

g. OT&E reports be made solely by the OT&E authority. 

h. OT&E phase reports be issued only, with no progress or interim reports, 
to allow full analysis and evaluation of the test results. 

i. the USN Operational Test Director type course be a prerequisite for key 
billets within the OT&E authority until the RAN develops enough 
knowledge and expertise to develop its own course. 
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j. the OT&E authority develop acquaint courses for other personnel with 
a need to understand OT&E. 

k. the OT&E authority deliver presentations regularly to Project Directors 
Courses, the ILS Management and Acquisition Course (ILSMAC), 
Defence management courses, and other appropriate acquisition 
courses, so to increase the awareness of OT&E in the RAN. 

l. a Warfare Officer be appointed to head the OT&E authority. 

m. establishing corporate membership of a suitable professional T&E 
organisation be investigated. 
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APPENDIX A: USN OT&E COURSE SYLLABUS 



The Commander, Operational Test and Evaluation Force (COMOPTEVFOR) 
conducts a four day Operational Test Directors (OTD) course designed to acquaint 
prospective Operational Test Directors and other interested personnel with the 
fundamentals of OT&E philosophy, terminology, policy and procedures. This 
course covers the following subject areas: 

• Acquisition Program Overview 

• Test and Evaluation Master Plan (TE,MP) 

• Operational Effectiveness Overview 

• Operational Suitability Overview 

• Logistic Supportability 

• Fleet Research and Development T&E Support 

• Operational Security 

• Managing Money 

• Weapons Systems Survivability 

• Threat Support 

• Test Planning 

• Software Intensive Systems 

• Modeling, Simulation and Analysis 

• Conducting the Tests 

• Lessons Learned 

• Standards of Conduct 

• Evaluation Reports 

• Director, OT&E Perspectives 

• Chief of Naval Operations Perspectives 

• Program Manager Perspectives 
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• Relationship with General Accounting Office 

• Test Resources 

• Program Formulation 
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APPENDIX B: ACRONYM LIST 



ABBREVIATION 


FULL TITLE 


A 




AAW 


Antiair Warfare 


AB 


Acceptance Board (RAN) 


ABR 


Australian Book of Reference (RAN) 


ACDEV-N 


Assistant Chief of Naval Staff (Development) 


ACMAT-N 


Assistant Chief of Naval Staff (Materiel) 


ADF 


Australian Defence Force (RAN) 


AF 


Australian Frigate (RAN) 


AINS 


Acceptance into Naval Service (RAN) 


AMP 


Acceptance Management Plan (RAN) 


ASC 


Agreed Ship Characteristics (RAN) 


ASUW 


Antisurface Warfare 


ASW 


Antisubmarine Warfare 


c 




CAP 


Combat Air Patrol 


CCP 


Chief of Capital Procurement (RAN) 


CEPMAN 1 


Capital Equipment Procurement Manual (RAN) 


CIWS 


Close-in Weapons System 


CNO 


Chief of Naval Operations (USN) 


CNS 


Chief of Naval Staff (RAN) 


COEA 


Cost and Operational Effectiveness Analysis 
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COI 


Critical Operational Issue 


COMFLOT 


Commodore Flotillas (RAN) 


COMNAVSEA 


Commander, Naval Sea Systems Command (USN) 


COMOPTEVFOR 


Commander, Operational Test and Evaluation 
Force (USN) 


CPO 


Capital Procurement Organisation (RAN) 


CSE 


Combat System Evaluation (RAN) 


CSO(E) 


Chief Staff Officer (Engineering) (RAN) 


CSO(O) 


Chief Staff Officer (Operations) (RAN) 


CSSQT 


Combat System Ship Qualification Trial (RAN) 


CST 


Combat System Trainer (RAN) 


CTE 


Commander Test and Evaluation (RAN) 


D 




DAA 


Design Approval Authority (RAN) 


DCNS 


Deputy Chief of Naval Staff (RAN) 


DDG 


Guided Missile Destroyer 


DFCP 


Defence Force Capability Proposal (RAN) 


DGFD(SEA) 


Director General Force Development (Sea) (RAN) 


DGNPW 


Director General Naval Policy and Warfare (RAN) 


DGNW 


Director General Naval Warfare (RAN) 


DI(N) 


Defence Instruction (Navy) (RAN) 


DLSIE 


Defence Logistics Studies Information Exchange 
(USN) 


DNUR 


Director of Naval User Requirements (RAN) 


DNW 


Director of Naval Warfare (RAN) 
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DoD 


Department of Defence 


DOT&E 


Director of Operational Test and Evaluation (USN) 


DSMC 


Defence Systems Management College (USN) 


DT&E 


Development Test and Evaluation 


E 




ECM 


Electronic Countermeasures 


ESM 


Electronic Surveillance Measures 


EW 


Electronic Warfare 


F 




FCT 


Final Contract Trials (USN) 


FFG 


Guided Missile Frigate 


FLEETEX 


Fleet Exercise (USN) 


FOT&E 


Follow-on Operational Test and Evaluation (USN) 


FSD 


Full Scale Development (USN) 


G 




GAO 


General Accounting Office (USN) 


GDP 


Gross Domestic Product 


H 




HATS 


Harbour Acceptance Trials (RAN) 


HMAS 


Her Majesty’s Australian Ship 


HQADF 


Headquarters, Australian Defence Force (RAN) 


I 




ILS 


Integrated Logistic Support 


ILSP 


Integrated Logistic Support Plan 


IOT&E 


Initial Operational Test and Evaluation (USN) 
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ITP 


Integrated Test Package (RAN) 


ITT 

f 


Inspections, Tests and Trials (RAN) 


L 

LBTS 


Land Based Test Site 


LRIP 


Low Rate Initial Production (USN) 


M 




MHQ 


Maritime Headquarters (RAN) 


MNS 


Mission Need Statement (USN) 


MOE 


Measures of Effectiveness 


N 




NAVSEA 


Naval Sea Systems Command (USN) 


NCP 


Naval Capability Proposal (RAN) 


NWAC 


Naval Weapons Analysis Centre (USN) 


0 




OA 


Operational Acceptance (RAN) 


OCP 


Operational Concept Paper (RAN) 


OPD 


Operational Performance Demonstration (RAN) 


OPEVAL 


Operational Evaluation 


OPTEVFOR 


Operational Test and Evaluation Force (USN) 


ORD 


Operational Requirements Document (USN) 


ORE 


Operational Readiness Evaluation (RAN) 


OSD 


Office of the Secretary of Defence (USN) 


OTA 


Operational Test and Evaluation Agency (USN) 


OTD 


Operational Test Director (USN) 


OT&E 


Operational Test and Evaluation 
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p 



PA 


Preliminary Acceptance (RAN) 


PAT 


Production Acceptance Test (RAN) 


PAT&E 


Production Acceptance Test and Evaluation 


PD 


Project Director (RAN) 


PDA 


Post Delivery Availability 


PDT&T 


Post Delivery Tests and Trials (RAN) 


PM 


Program Manager (USN) 


PMBS 


Program Management and Budgeting System 
(RAN) 


PREINSURV 


President, Board of Inspection and Survey (USN) 


PSA 


Post Shakedown Availability (USN) 


R 




R&D 


Research and Development 


RAN 


Royal Australian Navy 


RANTAU 


RAN Trials and Assessing Unit (RAN) 


REFTRA 


Refresher Training (USN) 


RM&A 


Reliability, Maintainability and Availability 


ROC 


Required Operational Characteristics (RAN) 


S 




SATS 


Sea Acceptance Trials (RAN) 


SECDEF 


Secretary of Defence 


SQT 


Ship Qualification Trial (RAN) 


STW 


Strike Warfare (USN) 


SOE 


Schedule of Events (USN) 
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SUPSHIPS 


Supervisor of Shipbuilding (USN) 


T 




T&E 


Test and Evaluation 


TECG 


Test and Evaluation Coordination Group (USN) 


TEMP 


Test and Evaluation Master Plan 


TEPG 


Test and Evaluation Planning Group (RAN) 


TLR 


Top Level Requirement (USN) 


TSTP 


Total Ship Test Program 


U 




USN 


United States Navy 


uss 


United States Ship 


w 




WSRL 


Weapons Systems Research Laboratory (RAN) 
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